Jul 28 10:07:20.804194 ip-10-0-133-108 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:07:21.140053 ip-10-0-133-108 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:21.140053 ip-10-0-133-108 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:07:21.140053 ip-10-0-133-108 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:21.140053 ip-10-0-133-108 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:07:21.140053 ip-10-0-133-108 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:07:21.140986 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.140895    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:07:21.147481 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147458    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.147481 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147477    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.147481 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147481    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.147481 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147484    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.147481 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147488    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147491    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147494    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147497    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147500    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147503    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147505    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147508    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147511    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147513    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147516    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147519    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147522    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147524    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147527    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147530    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147532    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147535    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147537    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147540    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.147685 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147544    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147548    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147551    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147554    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147556    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147560    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147563    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147566    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147575    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147578    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147581    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147583    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147586    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147589    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147592    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147595    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147597    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147600    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147603    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147605    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.148517 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147608    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147611    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147613    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147616    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147619    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147621    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147624    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147626    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147629    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147631    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147634    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147637    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147639    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147642    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147644    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147647    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147649    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147652    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147654    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147657    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.149075 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147659    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147662    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147669    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147672    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147676    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147680    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147686    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147689    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147692    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147695    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147699    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147702    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147704    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147707    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147710    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147713    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147716    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147719    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147723    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.149550 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147726    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147729    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147732    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147735    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147737    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147740    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147742    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147745    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147747    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147750    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147752    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147755    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147757    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147760    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147762    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.147765    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148471    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148478    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148481    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148484    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.150017 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148487    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148491    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148493    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148496    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148498    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148501    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148504    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148506    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148510    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148512    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148515    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148518    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148520    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148524    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148526    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148529    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148532    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148535    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148538    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.150498 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148540    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148544    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148547    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148549    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148552    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148554    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148557    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148561    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148565    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148568    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148571    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148574    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148577    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148579    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148582    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148585    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148587    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148590    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148592    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.150997 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148600    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148603    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148606    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148609    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148611    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148614    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148617    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148619    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148623    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148626    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148629    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148631    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148634    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148636    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148640    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148642    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148645    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148647    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148652    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.151469 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148655    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148657    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148660    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148662    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148665    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148668    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148671    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148674    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148676    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148679    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148681    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148684    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148686    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148689    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148691    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148694    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148696    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148699    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148701    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148703    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.151952 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148706    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148708    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148711    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148715    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148718    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148723    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148726    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148729    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148732    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148734    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148737    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148739    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148742    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148745    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148747    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148750    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148753    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.148756    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149148    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.152428 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149153    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149156    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149159    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149162    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149164    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149167    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149170    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149172    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149175    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149179    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149183    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149186    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149189    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149193    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149195    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149199    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149202    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149205    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149208    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.152899 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149211    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149213    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149216    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149219    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149221    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149224    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149229    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149231    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149234    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149236    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149239    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149241    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149244    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149246    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149249    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149251    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149254    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149257    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149259    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.153379 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149262    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149264    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149267    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149269    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149272    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149274    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149277    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149280    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149283    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149286    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149289    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149291    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149294    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149296    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149299    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149301    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149304    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149306    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149309    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149312    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.153838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149315    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149318    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149320    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149339    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149342    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149345    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149348    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149351    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149354    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149357    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149359    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149362    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149364    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149367    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149369    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149372    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149374    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149377    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149381    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149384    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.154356 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149386    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149389    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149392    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149395    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149397    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149400    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149402    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149405    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149407    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149410    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149412    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149417    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149421    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149424    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149427    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149429    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149432    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149434    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149437    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.149439    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.154838 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149502    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149509    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149516    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149521    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149526    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149531    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149537    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149543    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149547    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149550    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149554    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149558    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149561    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149564    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149567    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149570    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149574    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149577    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149582    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149584    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149587    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149590    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149594    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149599    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:07:21.155332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149602    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149605    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149608    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149611    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149614    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149617    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149621    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149624    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149628    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149632    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149634    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149637    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149641    2569 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149643    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149649    2569 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149652    2569 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149655    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149658    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149661    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149669    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149672    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149675    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149678    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149681    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149684    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:07:21.155957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149694    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149697    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149700    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149703    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149706    2569 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149714    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149719    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149722    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149725    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149728    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149731    2569 flags.go:64] FLAG: --help="false"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149734    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149738    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149740    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149743    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149747    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149750    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149753    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149756    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149759    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149762    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149764    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149767    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149770    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:07:21.156577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149773    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149776    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149780    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149783    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149786    2569 flags.go:64] FLAG: --lock-file=""
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149788    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149792    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149794    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149800    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149803    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149806    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149809    2569 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149811    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149815    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149818    2569 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149821    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149825    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149828    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149832    2569 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149835    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149838    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149841    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149844    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149847    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149850    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:07:21.157171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149852    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149860    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149863    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149866    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149869    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149872    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149877    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149880    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149883    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149887    2569 flags.go:64] FLAG: --port="10250"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149890    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149893    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d6c2936342ae984"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149897    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149900    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149902    2569 flags.go:64] FLAG: --register-node="true"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149905    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149909    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149912    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149915    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149918    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149942    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149947    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149951    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149955    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149958    2569 flags.go:64] FLAG: --runonce="false"
Jul 28 10:07:21.157784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149960    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149963    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149966    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149969    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149972    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149975    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149978    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149982    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149984    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149987    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149990    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149993    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149996    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.149999    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150002    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150007    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150011    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150014    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150018    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150021    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150024    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150027    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150030    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150033    2569 flags.go:64] FLAG: --v="2"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150038    2569 flags.go:64] FLAG: --version="false"
Jul 28 10:07:21.158405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150042    2569 flags.go:64] FLAG: --vmodule=""
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150046    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150049    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150160    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150164    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150168    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150170    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150173    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150176    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150180    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150182    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150185    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150187    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150190    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150192    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150195    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150197    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150200    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150202    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150205    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.159047 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150207    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150210    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150212    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150217    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150219    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150222    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150224    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150227    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150229    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150232    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150234    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150237    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150239    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150242    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150245    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150247    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150250    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150253    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150257    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150260    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.159545 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150263    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150266    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150268    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150271    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150275    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150278    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150281    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150284    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150287    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150289    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150292    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150295    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150298    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150300    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150303    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150307    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150309    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150312    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150314    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.160061 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150317    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150320    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150322    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150325    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150327    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150330    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150332    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150335    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150338    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150340    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150343    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150346    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150348    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150351    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150353    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150355    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150358    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150360    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150363    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150365    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.160525 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150368    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150370    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150373    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150375    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150378    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150380    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150384    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150387    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150392    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150394    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150397    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150400    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150402    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150405    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150408    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150410    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150413    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150416    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150418    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.161019 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150421    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150425    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150428    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.150430    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.150908    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157071    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157085    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157104    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157110    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157167    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157172    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157175    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157178    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157182    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157185    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.161479 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157188    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157191    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157194    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157197    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157199    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157203    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157205    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157208    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157211    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157213    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157216    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157219    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157221    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157224    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157227    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157230    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157232    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157234    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157243    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157247    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.161857 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157251    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157254    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157257    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157261    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157264    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157268    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157272    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157275    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157278    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157280    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157283    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157286    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157288    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157291    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157294    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157296    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157299    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157301    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157304    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.162355 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157307    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157309    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157312    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157314    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157317    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157320    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157322    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157325    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157327    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157330    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157332    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157335    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157338    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157341    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157344    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157348    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157351    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157354    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157357    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.162836 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157359    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157362    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157364    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157367    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157369    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157372    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157374    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157377    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157379    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157382    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157384    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157387    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157389    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157392    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157394    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157397    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157399    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157402    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157405    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157407    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.163313 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157410    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157412    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157415    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157417    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157420    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157422    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157425    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157427    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157430    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157433    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157436    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157440    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157442    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157445    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157447    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.163797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157452    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157575    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157580    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157584    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157587    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157591    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157593    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157596    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157599    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157602    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157606    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157608    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157611    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157613    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157615    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157619    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157622    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157625    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157627    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:07:21.164260 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157630    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157632    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157635    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157637    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157640    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157643    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157645    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157649    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157651    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157653    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157656    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157658    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157661    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157663    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157665    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157668    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157670    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157673    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157675    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157678    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:07:21.164710 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157680    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157682    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157685    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157687    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157690    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157692    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157695    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157697    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157700    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157702    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157704    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157707    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157709    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157712    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157714    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157717    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157719    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157722    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157725    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157728    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:07:21.165202 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157730    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157734    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157738    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157741    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157743    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157746    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157749    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157751    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157754    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157756    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157759    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157761    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157764    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157766    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157769    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157771    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157774    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157777    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157779    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:07:21.165683 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157782    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157784    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157787    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157789    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157792    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157794    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157798    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157801    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157804    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157807    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157810    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157812    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157816    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157818    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157833    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157837    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157840    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157843    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157845    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157848    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:07:21.166152 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157851    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:21.157854    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157858    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.157962    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.159765    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.161810    2569 server.go:1019] "Starting client certificate rotation"
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.161905    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.161955    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:07:21.166665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.164486    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:21.167074 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.167061    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:07:21.175915 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.175900    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:07:21.177775 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.177760    2569 log.go:25] "Validated CRI v1 image API"
Jul 28 10:07:21.178592 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.178570    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:07:21.180543 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.180521    2569 fs.go:135] Filesystem UUIDs: map[2b11d496-5c2a-4d4a-91fc-bada1e6cdbf6:/dev/nvme0n1p4 5706b171-32a1-4b13-aba6-5359b208e9c6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 10:07:21.180628 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.180542    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:07:21.185566 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.185449    2569 manager.go:216] Machine: {Timestamp:2026-07-28 10:07:21.184269693 +0000 UTC m=+0.275735335 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099913 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27459e96513b56a6e0209eed00d05b SystemUUID:ec27459e-9651-3b56-a6e0-209eed00d05b BootID:6cc5031f-6fc2-4b6c-b3a0-65eff7ff8853 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ee:ab:b1:89:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ee:ab:b1:89:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:15:02:90:c9:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:07:21.185566 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.185558    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:07:21.185724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.185659    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:07:21.188654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.188624    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:07:21.188784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.188657    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:07:21.189328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.189315    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:07:21.189328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.189328    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:07:21.189415 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.189335    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:21.189883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.189873    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:07:21.190811 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.190802    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:07:21.191164 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.191147    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:21.192600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192587    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:21.192748 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192737    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:07:21.192781 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192771    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:07:21.192810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192784    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:07:21.192810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192808    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:07:21.192863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192822    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:07:21.192863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.192834    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:07:21.193244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.193231    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:21.193306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.193251    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:07:21.195726 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.195711    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:07:21.197439 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.197425    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:07:21.197486 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.197455    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:07:21.198659 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198645    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:07:21.198694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198668    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:07:21.198694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198677    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:07:21.198694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198686    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198698    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198706    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198717    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198726    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198736    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198744    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198771    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:07:21.198799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.198781    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:07:21.199436 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.199426    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:07:21.199436 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.199436    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:07:21.202783 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.202770    2569 server.go:1268] "Started kubelet"
Jul 28 10:07:21.202894 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.202870    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:07:21.202998 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.202958    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:07:21.203042 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.203016    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:07:21.203974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.203948    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:07:21.204538 ip-10-0-133-108 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:07:21.205510 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.205487    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:07:21.207149 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.205626    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:07:21.207423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.207401    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:07:21.208237 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.208217    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:07:21.208965 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.208946    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:07:21.210557 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.208730    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-108.ec2.internal.18c66c2c9f1bd663  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-108.ec2.internal,UID:ip-10-0-133-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-108.ec2.internal,},FirstTimestamp:2026-07-28 10:07:21.202742883 +0000 UTC m=+0.294208525,LastTimestamp:2026-07-28 10:07:21.202742883 +0000 UTC m=+0.294208525,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-108.ec2.internal,}"
Jul 28 10:07:21.212346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212327    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:21.212413 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212333    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:07:21.212551 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.212533    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.212833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212804    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:07:21.212833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212806    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:07:21.212833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212832    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:07:21.213047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212982    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:07:21.213047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.212991    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:07:21.213643 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213612    2569 factory.go:55] Registering systemd factory
Jul 28 10:07:21.213643 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213627    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:07:21.213750 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213145    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgfct"
Jul 28 10:07:21.213944 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213915    2569 factory.go:153] Registering CRI-O factory
Jul 28 10:07:21.214003 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213949    2569 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:07:21.214003 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.213996    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:07:21.214113 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.214017    2569 factory.go:103] Registering Raw factory
Jul 28 10:07:21.214113 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.214032    2569 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:07:21.214458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.214440    2569 manager.go:349] Starting recovery of all containers
Jul 28 10:07:21.218324 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.218288    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:07:21.218398 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.218325    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:07:21.221281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.221262    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgfct"
Jul 28 10:07:21.224609 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.224594    2569 manager.go:354] Recovery completed
Jul 28 10:07:21.229244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.229232    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.231033 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231018    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.231094 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231050    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.231094 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231062    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.231601 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231586    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:07:21.231601 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231600    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:07:21.231713 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231590    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.231713 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.231618    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:07:21.232343 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232330    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.232394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232357    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.232394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232366    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.232964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232954    2569 policy_none.go:49] "None policy: Start"
Jul 28 10:07:21.232996 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232968    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:07:21.232996 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.232978    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:07:21.233599 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.233506    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-108.ec2.internal.18c66c2ca0cb81f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-108.ec2.internal,UID:ip-10-0-133-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-108.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-108.ec2.internal,},FirstTimestamp:2026-07-28 10:07:21.231032818 +0000 UTC m=+0.322498461,LastTimestamp:2026-07-28 10:07:21.231032818 +0000 UTC m=+0.322498461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-108.ec2.internal,}"
Jul 28 10:07:21.234149 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.234135    2569 policy_none.go:47] "Start"
Jul 28 10:07:21.274694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.274553    2569 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:07:21.274827 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.274714    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:07:21.274827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.274727    2569 server.go:87] "Starting device plugin registration server"
Jul 28 10:07:21.274968 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.274957    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:07:21.275009 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.274969    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:07:21.275083 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.275050    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:07:21.275169 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.275157    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:07:21.275234 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.275170    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:07:21.275643 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.275623    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:07:21.275730 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.275669    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.375088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.375059    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.376046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.376028    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.376109 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.376060    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.376109 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.376074    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.376109 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.376101    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.383722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.383705    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.383829 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.383726    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-108.ec2.internal\": node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.388425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.388396    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:07:21.389623 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.389604    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:07:21.389706 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.389637    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:07:21.389706 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.389658    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:07:21.389706 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.389692    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:07:21.391385 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.391333    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:21.396800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.396782    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.490549 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.490501    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"]
Jul 28 10:07:21.490676 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.490589    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.491594 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.491575    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.491697 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.491611    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.491697 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.491625    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.492836 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.492821    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.492956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.492941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.493011 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.492971    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.493518 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493503    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.493578 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493531    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.493578 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493543    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.493578 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493548    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.493578 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493568    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.493578 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.493578    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.494624 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.494611    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.494669 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.494636    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:07:21.495836 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.495818    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:07:21.495937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.495856    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:07:21.495937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.495871    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:07:21.496847 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.496831    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.514613 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.514587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.514613 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.514612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.514737 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.514630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45b79e16bf677a5e3c62a28e1b4ebba5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-108.ec2.internal\" (UID: \"45b79e16bf677a5e3c62a28e1b4ebba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.517536 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.517518    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-108.ec2.internal\" not found" node="ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.521709 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.521696    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-108.ec2.internal\" not found" node="ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.597746 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:21.597720    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:21.615721 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615697    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.615798 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.615798 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45b79e16bf677a5e3c62a28e1b4ebba5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-108.ec2.internal\" (UID: \"45b79e16bf677a5e3c62a28e1b4ebba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.615886 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615803    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.615886 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615804    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90e757adbfeaa0159630c0488b5430d2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal\" (UID: \"90e757adbfeaa0159630c0488b5430d2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:21.615886 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:21.615833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45b79e16bf677a5e3c62a28e1b4ebba5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-108.ec2.internal\" (UID: \"45b79e16bf677a5e3c62a28e1b4ebba5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:22.868950 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:22.868905    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:22.969423 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:22.969378    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:22.990919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:22.990897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:22.995493 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:22.995476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:23.070483 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:23.070398    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:23.170969 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:23.170937    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:23.271779 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:23.271753    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:23.331240 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.331178    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:07:23.331635 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.331313    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:07:23.360282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.360259    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:23.372730 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:23.372712    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-108.ec2.internal\" not found"
Jul 28 10:07:23.397662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.397632    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:02:22 +0000 UTC" deadline="2028-01-04 15:03:21.145292519 +0000 UTC"
Jul 28 10:07:23.397662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.397658    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12604h55m57.747637483s"
Jul 28 10:07:23.403974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.403960    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:23.483944 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.483910    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:23.495669 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.495647    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:23.497297 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.497284    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"
Jul 28 10:07:23.506664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.506642    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:07:23.603422 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:23.603385    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b79e16bf677a5e3c62a28e1b4ebba5.slice/crio-00e3624e9e40aa41ea30513901d6980cc9361878433e04f1a61bdd9fd2578c8e WatchSource:0}: Error finding container 00e3624e9e40aa41ea30513901d6980cc9361878433e04f1a61bdd9fd2578c8e: Status 404 returned error can't find the container with id 00e3624e9e40aa41ea30513901d6980cc9361878433e04f1a61bdd9fd2578c8e
Jul 28 10:07:23.607197 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.607183    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:07:23.710751 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:23.710723    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e757adbfeaa0159630c0488b5430d2.slice/crio-588143b28ab7f3c686a9b17da604d1efc594073b07d3a674781cc27f9a5b9983 WatchSource:0}: Error finding container 588143b28ab7f3c686a9b17da604d1efc594073b07d3a674781cc27f9a5b9983: Status 404 returned error can't find the container with id 588143b28ab7f3c686a9b17da604d1efc594073b07d3a674781cc27f9a5b9983
Jul 28 10:07:23.775274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:23.775243    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:24.364873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.364837    2569 apiserver.go:52] "Watching apiserver"
Jul 28 10:07:24.367675 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.367652    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:07:24.372978 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.372956    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:07:24.373413 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.373386    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx","openshift-cluster-node-tuning-operator/tuned-l5m7h","openshift-dns/node-resolver-hg6wf","openshift-image-registry/node-ca-wdjkk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal","openshift-multus/multus-additional-cni-plugins-dgh6v","openshift-network-diagnostics/network-check-target-4rm74","kube-system/global-pull-secret-syncer-6k7q6","kube-system/konnectivity-agent-vf62c","openshift-multus/multus-jt8td","openshift-multus/network-metrics-daemon-2r6mj","openshift-network-operator/iptables-alerter-42b4h","openshift-ovn-kubernetes/ovnkube-node-kvnkg","kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal"]
Jul 28 10:07:24.376116 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.376093    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:24.376211 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.376188    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:24.378264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.378246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.378360 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.378332    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:24.380375 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.380351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.382835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.382813    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.382921 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.382844    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwrwl\""
Jul 28 10:07:24.382921 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.382815    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.383484 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.383468    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.383606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.383587    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:07:24.385587 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.385557    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.385704 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.385619    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:24.385704 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.385643    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.385704 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.385673    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b4ltr\""
Jul 28 10:07:24.385956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.385935    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:07:24.386376 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.386153    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.388027 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.387788    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.389956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.389935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.390059 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.389981    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.390059 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.390049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:07:24.390355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.390335    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:07:24.390432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.390397    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:07:24.390432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.390412    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.390636 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.390618    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hk6g9\""
Jul 28 10:07:24.392217 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.392188    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:07:24.392345 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.392330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.392955 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.392935    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:07:24.393050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.393001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xs55g\""
Jul 28 10:07:24.394724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.394702    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:07:24.394822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.394810    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:07:24.395043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395026    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:07:24.395626 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395c1740-3932-4e87-bf52-06ee0acdd2b3-hosts-file\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.395740 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395646    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82zc\" (UniqueName: \"kubernetes.io/projected/395c1740-3932-4e87-bf52-06ee0acdd2b3-kube-api-access-d82zc\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.395740 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395680    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04758362-69bc-4f6f-a3e5-8ff5d2373d57-serviceca\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjz2c\" (UniqueName: \"kubernetes.io/projected/04758362-69bc-4f6f-a3e5-8ff5d2373d57-kube-api-access-sjz2c\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-dbus\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395857    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:07:24.395876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vz4cj\""
Jul 28 10:07:24.396141 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.395869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04758362-69bc-4f6f-a3e5-8ff5d2373d57-host\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.396196 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396145    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.396245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396204    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-kubelet-config\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c1740-3932-4e87-bf52-06ee0acdd2b3-tmp-dir\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vjd\" (UniqueName: \"kubernetes.io/projected/46632d46-0a7f-4b86-99bd-65ad313076e8-kube-api-access-d2vjd\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.397296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.396867    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:07:24.398351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.398333    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.398619 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.398591    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.398952 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.398934    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.399054 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.399011    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5sf5p\""
Jul 28 10:07:24.400763 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.400744    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:07:24.400852 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.400801    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-njdjf\""
Jul 28 10:07:24.401094 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.401077    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.403418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.403193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.403418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.403193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.403418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.403354    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ld7dr\""
Jul 28 10:07:24.403418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.403418    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:07:24.403657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.403478    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.405598 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.405578    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:07:24.405778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.405764    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:07:24.405844 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.405810    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:07:24.406401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.406385    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l2vmd\""
Jul 28 10:07:24.414920 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.414899    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hn5k"
Jul 28 10:07:24.422172 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.422133    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7hn5k"
Jul 28 10:07:24.484322 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.484300    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:07:24.497038 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395c1740-3932-4e87-bf52-06ee0acdd2b3-hosts-file\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82zc\" (UniqueName: \"kubernetes.io/projected/395c1740-3932-4e87-bf52-06ee0acdd2b3-kube-api-access-d82zc\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjz2c\" (UniqueName: \"kubernetes.io/projected/04758362-69bc-4f6f-a3e5-8ff5d2373d57-kube-api-access-sjz2c\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-konnectivity-ca\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-config\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497139    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395c1740-3932-4e87-bf52-06ee0acdd2b3-hosts-file\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.497189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tqf\" (UniqueName: \"kubernetes.io/projected/9a2d1abe-36a3-4e30-a01e-a238eba3decb-kube-api-access-n8tqf\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497198    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-multus-daemon-config\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cddee4ee-e652-477b-838b-b04d06767bff-iptables-alerter-script\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497256    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-system-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-etc-kubernetes\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c1740-3932-4e87-bf52-06ee0acdd2b3-tmp-dir\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-agent-certs\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-conf\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-kubelet\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.497520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-env-overrides\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vjd\" (UniqueName: \"kubernetes.io/projected/46632d46-0a7f-4b86-99bd-65ad313076e8-kube-api-access-d2vjd\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-slash\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-hostroot\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cnibin\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-etc-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c1740-3932-4e87-bf52-06ee0acdd2b3-tmp-dir\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04758362-69bc-4f6f-a3e5-8ff5d2373d57-host\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-kubelet-config\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497819    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-node-log\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-bin\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-var-lib-kubelet\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04758362-69bc-4f6f-a3e5-8ff5d2373d57-host\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-kubelet-config\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.498000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-device-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-kubelet\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovn-node-metrics-cert\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysconfig\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.497990    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-tuned\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6cmz\" (UniqueName: \"kubernetes.io/projected/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-kube-api-access-h6cmz\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-k8s-cni-cncf-io\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498075    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-sys-fs\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-netns\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-multus-certs\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.498146    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04758362-69bc-4f6f-a3e5-8ff5d2373d57-serviceca\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-var-lib-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.498212    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.498243    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:24.998204344 +0000 UTC m=+2.918908386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:24.498925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498280    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-run\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.498290    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:24.998274021 +0000 UTC m=+2.918978065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498317    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476bw\" (UniqueName: \"kubernetes.io/projected/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-kube-api-access-476bw\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-dbus\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-systemd-units\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-cnibin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498516    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-conf-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-socket-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-modprobe-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-systemd\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04758362-69bc-4f6f-a3e5-8ff5d2373d57-serviceca\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498607    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-dbus\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-sys\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.499715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-tmp\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498714    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-cni-binary-copy\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-host\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498784    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cddee4ee-e652-477b-838b-b04d06767bff-host-slash\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2zr\" (UniqueName: \"kubernetes.io/projected/cddee4ee-e652-477b-838b-b04d06767bff-kube-api-access-mn2zr\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498846    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-ovn\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-multus\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-netns\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-kubernetes\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498951    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-lib-modules\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbskp\" (UniqueName: \"kubernetes.io/projected/cb29c0a5-1710-4991-a3dc-65c32666afd5-kube-api-access-qbskp\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.498993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-system-cni-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-os-release\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.500490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-log-socket\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-script-lib\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499087    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499106    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-bin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvgd\" (UniqueName: \"kubernetes.io/projected/35d00954-6525-4b04-937c-b49920e44a34-kube-api-access-vvvgd\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499143    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-systemd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-netd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-socket-dir-parent\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-os-release\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499235    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-registration-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.501286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.499294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.505171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.505071    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:07:24.505377 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.505355    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:24.505377 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.505380    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:24.505545 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.505402    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:24.505545 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:24.505497    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:25.005479802 +0000 UTC m=+2.926183832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:24.508881 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.508860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjz2c\" (UniqueName: \"kubernetes.io/projected/04758362-69bc-4f6f-a3e5-8ff5d2373d57-kube-api-access-sjz2c\") pod \"node-ca-wdjkk\" (UID: \"04758362-69bc-4f6f-a3e5-8ff5d2373d57\") " pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.508988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.508900    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82zc\" (UniqueName: \"kubernetes.io/projected/395c1740-3932-4e87-bf52-06ee0acdd2b3-kube-api-access-d82zc\") pod \"node-resolver-hg6wf\" (UID: \"395c1740-3932-4e87-bf52-06ee0acdd2b3\") " pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.508988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.508911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vjd\" (UniqueName: \"kubernetes.io/projected/46632d46-0a7f-4b86-99bd-65ad313076e8-kube-api-access-d2vjd\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:24.564591 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.564545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal" event={"ID":"90e757adbfeaa0159630c0488b5430d2","Type":"ContainerStarted","Data":"588143b28ab7f3c686a9b17da604d1efc594073b07d3a674781cc27f9a5b9983"}
Jul 28 10:07:24.565722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.565679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal" event={"ID":"45b79e16bf677a5e3c62a28e1b4ebba5","Type":"ContainerStarted","Data":"00e3624e9e40aa41ea30513901d6980cc9361878433e04f1a61bdd9fd2578c8e"}
Jul 28 10:07:24.599972 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.599941    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.599976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-host\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cddee4ee-e652-477b-838b-b04d06767bff-host-slash\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.600121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2zr\" (UniqueName: \"kubernetes.io/projected/cddee4ee-e652-477b-838b-b04d06767bff-kube-api-access-mn2zr\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.600121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cddee4ee-e652-477b-838b-b04d06767bff-host-slash\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.600121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600079    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-host\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600140    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-ovn\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-multus\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-multus\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600261    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600273    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-ovn\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-netns\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-kubernetes\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-lib-modules\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-netns\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbskp\" (UniqueName: \"kubernetes.io/projected/cb29c0a5-1710-4991-a3dc-65c32666afd5-kube-api-access-qbskp\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-system-cni-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-kubernetes\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-os-release\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-system-cni-dir\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-log-socket\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-os-release\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-lib-modules\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-script-lib\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600539    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-log-socket\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-bin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvgd\" (UniqueName: \"kubernetes.io/projected/35d00954-6525-4b04-937c-b49920e44a34-kube-api-access-vvvgd\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-systemd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-netd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.600883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-cni-bin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600649    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-socket-dir-parent\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-netd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600739    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-socket-dir-parent\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-run-systemd\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600852    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-os-release\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-registration-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-os-release\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-konnectivity-ca\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.600994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-config\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-registration-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tqf\" (UniqueName: \"kubernetes.io/projected/9a2d1abe-36a3-4e30-a01e-a238eba3decb-kube-api-access-n8tqf\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601037    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601055    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-multus-daemon-config\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cddee4ee-e652-477b-838b-b04d06767bff-iptables-alerter-script\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.601672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-system-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-etc-kubernetes\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-script-lib\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601204    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-agent-certs\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601246    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-conf\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-kubelet\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-env-overrides\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-slash\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-hostroot\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cnibin\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-etc-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601452    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.602475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601484    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-conf\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-node-log\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-bin\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-var-lib-kubelet\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601548    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovnkube-config\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-konnectivity-ca\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601587    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-device-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-etc-kubernetes\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-etc-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601652    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-var-lib-kubelet\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cnibin\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-cni-bin\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601715    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-node-log\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-var-lib-kubelet\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601749    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-slash\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601772    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-hostroot\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-device-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601803    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-kubelet\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovn-node-metrics-cert\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-multus-daemon-config\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601857    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysconfig\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-tuned\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601902    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysconfig\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cmz\" (UniqueName: \"kubernetes.io/projected/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-kube-api-access-h6cmz\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-system-cni-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-k8s-cni-cncf-io\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-kubelet\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-sys-fs\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602004    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-netns\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cddee4ee-e652-477b-838b-b04d06767bff-iptables-alerter-script\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.601857    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-multus-certs\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-multus-certs\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-var-lib-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.603938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602081    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a2d1abe-36a3-4e30-a01e-a238eba3decb-env-overrides\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-run\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476bw\" (UniqueName: \"kubernetes.io/projected/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-kube-api-access-476bw\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-var-lib-openvswitch\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602163    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-netns\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-host-run-k8s-cni-cncf-io\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-systemd-units\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-sys-fs\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-run\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-systemd-units\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a2d1abe-36a3-4e30-a01e-a238eba3decb-host-run-ovn-kubernetes\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-cnibin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-conf-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602414    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-multus-conf-dir\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35d00954-6525-4b04-937c-b49920e44a34-cnibin\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.604618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-socket-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-modprobe-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-sysctl-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-systemd\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602512    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-sys\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-tmp\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-cni-binary-copy\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-modprobe-d\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb29c0a5-1710-4991-a3dc-65c32666afd5-socket-dir\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-sys\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.602732    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-systemd\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.603049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35d00954-6525-4b04-937c-b49920e44a34-cni-binary-copy\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.604376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef-agent-certs\") pod \"konnectivity-agent-vf62c\" (UID: \"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef\") " pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.604554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-etc-tuned\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.604910    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-tmp\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.605355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.604917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a2d1abe-36a3-4e30-a01e-a238eba3decb-ovn-node-metrics-cert\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.609148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.609122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tqf\" (UniqueName: \"kubernetes.io/projected/9a2d1abe-36a3-4e30-a01e-a238eba3decb-kube-api-access-n8tqf\") pod \"ovnkube-node-kvnkg\" (UID: \"9a2d1abe-36a3-4e30-a01e-a238eba3decb\") " pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.609577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.609554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbskp\" (UniqueName: \"kubernetes.io/projected/cb29c0a5-1710-4991-a3dc-65c32666afd5-kube-api-access-qbskp\") pod \"aws-ebs-csi-driver-node-dfqdx\" (UID: \"cb29c0a5-1710-4991-a3dc-65c32666afd5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.609784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.609765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2zr\" (UniqueName: \"kubernetes.io/projected/cddee4ee-e652-477b-838b-b04d06767bff-kube-api-access-mn2zr\") pod \"iptables-alerter-42b4h\" (UID: \"cddee4ee-e652-477b-838b-b04d06767bff\") " pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.610114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.610097    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvgd\" (UniqueName: \"kubernetes.io/projected/35d00954-6525-4b04-937c-b49920e44a34-kube-api-access-vvvgd\") pod \"multus-jt8td\" (UID: \"35d00954-6525-4b04-937c-b49920e44a34\") " pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.610328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.610312    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cmz\" (UniqueName: \"kubernetes.io/projected/e954e4b3-8d05-41ee-b3fc-12e312fb6e97-kube-api-access-h6cmz\") pod \"tuned-l5m7h\" (UID: \"e954e4b3-8d05-41ee-b3fc-12e312fb6e97\") " pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.611455 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.611434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-476bw\" (UniqueName: \"kubernetes.io/projected/1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612-kube-api-access-476bw\") pod \"multus-additional-cni-plugins-dgh6v\" (UID: \"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612\") " pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.691400 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.691331    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg6wf"
Jul 28 10:07:24.699951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.699928    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wdjkk"
Jul 28 10:07:24.707936 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.707918    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgh6v"
Jul 28 10:07:24.712751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.712679    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:24.719624 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.719603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:24.725452 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.725429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l5m7h"
Jul 28 10:07:24.732090 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.732067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jt8td"
Jul 28 10:07:24.738763 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.738744    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx"
Jul 28 10:07:24.745368 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:24.745350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-42b4h"
Jul 28 10:07:24.941342 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.941307    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04758362_69bc_4f6f_a3e5_8ff5d2373d57.slice/crio-7b76e2ca8f46d9280c0a6464ec83816dce4fcef92e736bde0ee1a0503f9272dd WatchSource:0}: Error finding container 7b76e2ca8f46d9280c0a6464ec83816dce4fcef92e736bde0ee1a0503f9272dd: Status 404 returned error can't find the container with id 7b76e2ca8f46d9280c0a6464ec83816dce4fcef92e736bde0ee1a0503f9272dd
Jul 28 10:07:24.942824 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.942795    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395c1740_3932_4e87_bf52_06ee0acdd2b3.slice/crio-4f402bb27bba292451bcb3f7ba97ee858e5943358bd02212db081f66035e0741 WatchSource:0}: Error finding container 4f402bb27bba292451bcb3f7ba97ee858e5943358bd02212db081f66035e0741: Status 404 returned error can't find the container with id 4f402bb27bba292451bcb3f7ba97ee858e5943358bd02212db081f66035e0741
Jul 28 10:07:24.946096 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.946071    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb29c0a5_1710_4991_a3dc_65c32666afd5.slice/crio-466734a745e0d6d0b338d6c6ffb53d7090a2571a4d33a590627c2157990feb57 WatchSource:0}: Error finding container 466734a745e0d6d0b338d6c6ffb53d7090a2571a4d33a590627c2157990feb57: Status 404 returned error can't find the container with id 466734a745e0d6d0b338d6c6ffb53d7090a2571a4d33a590627c2157990feb57
Jul 28 10:07:24.947724 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.947664    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ab969_d6e3_4f7b_9e3b_5ed7dab44cef.slice/crio-5a26919ed020cce451990de8122bce4fb31477945b612ff4707fe253cbcf22d8 WatchSource:0}: Error finding container 5a26919ed020cce451990de8122bce4fb31477945b612ff4707fe253cbcf22d8: Status 404 returned error can't find the container with id 5a26919ed020cce451990de8122bce4fb31477945b612ff4707fe253cbcf22d8
Jul 28 10:07:24.948417 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.948330    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d00954_6525_4b04_937c_b49920e44a34.slice/crio-5a9cc00b66d6b066f303f3947569668da13b8860b81000c47bd7a1e0b68c1512 WatchSource:0}: Error finding container 5a9cc00b66d6b066f303f3947569668da13b8860b81000c47bd7a1e0b68c1512: Status 404 returned error can't find the container with id 5a9cc00b66d6b066f303f3947569668da13b8860b81000c47bd7a1e0b68c1512
Jul 28 10:07:24.949205 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.949179    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaa1fdd_6ff4_4f47_9d3b_d6d00b17c612.slice/crio-f7691baa0b839f0153322bc823cd13b068ef1ce642c8228318441ba9b042b0f8 WatchSource:0}: Error finding container f7691baa0b839f0153322bc823cd13b068ef1ce642c8228318441ba9b042b0f8: Status 404 returned error can't find the container with id f7691baa0b839f0153322bc823cd13b068ef1ce642c8228318441ba9b042b0f8
Jul 28 10:07:24.950218 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.950005    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode954e4b3_8d05_41ee_b3fc_12e312fb6e97.slice/crio-9e6221d3ab347f16bbc4571a884a2edfab58ffb0b3a9172d889dcfb098940b32 WatchSource:0}: Error finding container 9e6221d3ab347f16bbc4571a884a2edfab58ffb0b3a9172d889dcfb098940b32: Status 404 returned error can't find the container with id 9e6221d3ab347f16bbc4571a884a2edfab58ffb0b3a9172d889dcfb098940b32
Jul 28 10:07:24.972800 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.972780    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2d1abe_36a3_4e30_a01e_a238eba3decb.slice/crio-6efdb163f0014dc4c9ff9e744caac8ba6eabea5f740e3432403dcfb69313526a WatchSource:0}: Error finding container 6efdb163f0014dc4c9ff9e744caac8ba6eabea5f740e3432403dcfb69313526a: Status 404 returned error can't find the container with id 6efdb163f0014dc4c9ff9e744caac8ba6eabea5f740e3432403dcfb69313526a
Jul 28 10:07:24.973608 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:07:24.973580    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddee4ee_e652_477b_838b_b04d06767bff.slice/crio-acc59617259faf12731b9408a83e0595fbcdcd17de871604e6fc274204755d9a WatchSource:0}: Error finding container acc59617259faf12731b9408a83e0595fbcdcd17de871604e6fc274204755d9a: Status 404 returned error can't find the container with id acc59617259faf12731b9408a83e0595fbcdcd17de871604e6fc274204755d9a
Jul 28 10:07:25.005567 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.005541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:25.005671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.005586    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:25.005671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.005622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:25.005800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005729    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:25.005800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005765    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:25.005800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005784    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:25.005800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005783    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:25.005800 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005795    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:25.005990 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005843    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:26.0058212 +0000 UTC m=+3.926525250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:25.005990 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005864    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:26.005853728 +0000 UTC m=+3.926557764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:25.005990 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.005881    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:26.005871964 +0000 UTC m=+3.926575998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:25.425501 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.425328    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:24 +0000 UTC" deadline="2028-01-21 07:48:30.314999151 +0000 UTC"
Jul 28 10:07:25.425501 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.425386    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13005h41m4.889617061s"
Jul 28 10:07:25.561856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.561227    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:25.561856 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.561359    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:25.561856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.561438    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:25.561856 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:25.561505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:25.573925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.573892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42b4h" event={"ID":"cddee4ee-e652-477b-838b-b04d06767bff","Type":"ContainerStarted","Data":"acc59617259faf12731b9408a83e0595fbcdcd17de871604e6fc274204755d9a"}
Jul 28 10:07:25.577189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.577113    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l5m7h" event={"ID":"e954e4b3-8d05-41ee-b3fc-12e312fb6e97","Type":"ContainerStarted","Data":"9e6221d3ab347f16bbc4571a884a2edfab58ffb0b3a9172d889dcfb098940b32"}
Jul 28 10:07:25.579869 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.579804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf62c" event={"ID":"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef","Type":"ContainerStarted","Data":"5a26919ed020cce451990de8122bce4fb31477945b612ff4707fe253cbcf22d8"}
Jul 28 10:07:25.584114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.584038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx" event={"ID":"cb29c0a5-1710-4991-a3dc-65c32666afd5","Type":"ContainerStarted","Data":"466734a745e0d6d0b338d6c6ffb53d7090a2571a4d33a590627c2157990feb57"}
Jul 28 10:07:25.585744 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.585667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg6wf" event={"ID":"395c1740-3932-4e87-bf52-06ee0acdd2b3","Type":"ContainerStarted","Data":"4f402bb27bba292451bcb3f7ba97ee858e5943358bd02212db081f66035e0741"}
Jul 28 10:07:25.588623 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.588558    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wdjkk" event={"ID":"04758362-69bc-4f6f-a3e5-8ff5d2373d57","Type":"ContainerStarted","Data":"7b76e2ca8f46d9280c0a6464ec83816dce4fcef92e736bde0ee1a0503f9272dd"}
Jul 28 10:07:25.593014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.592959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal" event={"ID":"45b79e16bf677a5e3c62a28e1b4ebba5","Type":"ContainerStarted","Data":"544b5421b84ac8275e006fd3657977880480afa784eebc76219d1ec77a986ecb"}
Jul 28 10:07:25.597427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.597401    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"6efdb163f0014dc4c9ff9e744caac8ba6eabea5f740e3432403dcfb69313526a"}
Jul 28 10:07:25.602664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.602636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerStarted","Data":"f7691baa0b839f0153322bc823cd13b068ef1ce642c8228318441ba9b042b0f8"}
Jul 28 10:07:25.611146 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:25.611103    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt8td" event={"ID":"35d00954-6525-4b04-937c-b49920e44a34","Type":"ContainerStarted","Data":"5a9cc00b66d6b066f303f3947569668da13b8860b81000c47bd7a1e0b68c1512"}
Jul 28 10:07:26.011372 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.011336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:26.011579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.011389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:26.011579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.011451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:26.011721 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011581    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:26.011721 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011644    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:28.011623821 +0000 UTC m=+5.932327851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011855    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011881    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011894    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011890    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011938    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:28.011923521 +0000 UTC m=+5.932627565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:26.011974 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.011956    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:28.011947061 +0000 UTC m=+5.932651103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:26.426447 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.426354    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:24 +0000 UTC" deadline="2028-03-14 20:16:09.848583026 +0000 UTC"
Jul 28 10:07:26.426447 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.426398    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14290h8m43.422189836s"
Jul 28 10:07:26.562657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:26.562133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:26.562657 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:26.562277    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:27.561878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:27.561618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:27.562324 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:27.561991    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:27.562381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:27.562350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:27.562483 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:27.562463    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:27.630829 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:27.630789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal" event={"ID":"90e757adbfeaa0159630c0488b5430d2","Type":"ContainerStarted","Data":"ce5cb0e1f1c7ce0fceb4fac1c1b03d5a60f573d28490be867329141b2e60f819"}
Jul 28 10:07:27.650045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:27.649980    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-108.ec2.internal" podStartSLOduration=4.649952023 podStartE2EDuration="4.649952023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:25.608382271 +0000 UTC m=+3.529086325" watchObservedRunningTime="2026-07-28 10:07:27.649952023 +0000 UTC m=+5.570656076"
Jul 28 10:07:28.027100 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.027066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:28.027100 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.027121    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:28.027395 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.027152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:28.027395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027312    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:28.027395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027328    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:28.027395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027340    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:28.027395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027394    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:32.02737767 +0000 UTC m=+9.948081714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:28.027961 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027801    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:28.027961 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027853    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:32.027836541 +0000 UTC m=+9.948540574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:28.027961 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027903    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:28.027961 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.027932    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:32.027921876 +0000 UTC m=+9.948625920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:28.563201 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.563074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:28.563741 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:28.563223    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:28.635427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.635391    2569 generic.go:356] "Generic (PLEG): container finished" podID="90e757adbfeaa0159630c0488b5430d2" containerID="ce5cb0e1f1c7ce0fceb4fac1c1b03d5a60f573d28490be867329141b2e60f819" exitCode=0
Jul 28 10:07:28.635630 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:28.635437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal" event={"ID":"90e757adbfeaa0159630c0488b5430d2","Type":"ContainerDied","Data":"ce5cb0e1f1c7ce0fceb4fac1c1b03d5a60f573d28490be867329141b2e60f819"}
Jul 28 10:07:29.561528 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:29.561476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:29.561742 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:29.561597    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:29.561742 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:29.561476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:29.561742 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:29.561713    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:30.561268 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:30.560939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:30.561720 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:30.561400    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:31.560960 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:31.560915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:31.561147 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:31.561052    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:31.561147 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:31.561135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:31.561282 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:31.561258    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:32.064215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:32.064275    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:32.064308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064403    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064463    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064491    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.064467668 +0000 UTC m=+17.985171713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064531    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064541    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.064524278 +0000 UTC m=+17.985228330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064549    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064563    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:32.064663 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.064614    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.064601138 +0000 UTC m=+17.985305169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:32.561738 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:32.561645    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:32.562123 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:32.561788    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:33.561546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:33.560956    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:33.561546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:33.560986    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:33.561546 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:33.561101    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:33.561546 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:33.561241    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:34.562101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:34.561587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:34.562101 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:34.561746    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:35.560857 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:35.560824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:35.560981 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:35.560949    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:35.561042 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:35.561001    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:35.561144 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:35.561117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:36.560933 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:36.560893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:36.561342 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:36.561014    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:37.561538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:37.561499    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:37.561982 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:37.561513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:37.561982 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:37.561732    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:37.561982 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:37.561602    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:38.561494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:38.561450    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:38.561677 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:38.561595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:39.561468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:39.561385    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:39.561707 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:39.561381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:39.561707 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:39.561519    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:39.561707 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:39.561604    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:40.129541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:40.129508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:40.129723 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:40.129556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:40.129723 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:40.129621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:40.129723 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129649    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129741    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:56.129716876 +0000 UTC m=+34.050420918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129749    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129775    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129795    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129800    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:56.129783067 +0000 UTC m=+34.050487115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129808    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:40.129903 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.129851    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:56.129837481 +0000 UTC m=+34.050541526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:40.561680 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:40.561606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:40.561835 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:40.561764    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:41.561227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:41.561190    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:41.561227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:41.561220    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:41.561472 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:41.561309    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:41.561472 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:41.561457    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:42.562133 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:42.562100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:42.562603 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:42.562230    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:43.561572 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.560968    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:43.561572 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:43.561088    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:43.561572 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.561159    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:43.561572 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:43.561229    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:43.686108 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.685874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal" event={"ID":"90e757adbfeaa0159630c0488b5430d2","Type":"ContainerStarted","Data":"e207b0bd78fa4a8be6b48c0110cfd9de1df3b7a058719efbe47d4224c3a0bc1d"}
Jul 28 10:07:43.687290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.687269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l5m7h" event={"ID":"e954e4b3-8d05-41ee-b3fc-12e312fb6e97","Type":"ContainerStarted","Data":"1b6b6b7f31a8012750fcf64de925854b983bcbb87eb0d9ceaa9119216c96cd53"}
Jul 28 10:07:43.688397 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.688375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vf62c" event={"ID":"f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef","Type":"ContainerStarted","Data":"43ba5a31fa4b9f69a862100bd2f2103f70d053b0eaef84b343d668290efc90a6"}
Jul 28 10:07:43.689809 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.689787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx" event={"ID":"cb29c0a5-1710-4991-a3dc-65c32666afd5","Type":"ContainerStarted","Data":"128bd1f2d2afa532f819d5a1aff89abac77f425a83a226031afab57efcae3887"}
Jul 28 10:07:43.691351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.691308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg6wf" event={"ID":"395c1740-3932-4e87-bf52-06ee0acdd2b3","Type":"ContainerStarted","Data":"b00e0b110083ae34e8e9ef81dd675581fb09a307de6efca3ff3e27ff78db7bbb"}
Jul 28 10:07:43.692884 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.692862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wdjkk" event={"ID":"04758362-69bc-4f6f-a3e5-8ff5d2373d57","Type":"ContainerStarted","Data":"0658e380c77d9103611eb59e97d054a7999a32d96b69c01317093e0e262249f1"}
Jul 28 10:07:43.694551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.694508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerStarted","Data":"c8ed39e84d3927e393a7baf8ad6e4da2f0c7df1c18aa82bbe05c38f5bb01df2f"}
Jul 28 10:07:43.696045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.696014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jt8td" event={"ID":"35d00954-6525-4b04-937c-b49920e44a34","Type":"ContainerStarted","Data":"a99c8ff7da2a8511421e625ab481c0ef07d718eddaa88eb3a542860b4246cfdc"}
Jul 28 10:07:43.707795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.707520    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-108.ec2.internal" podStartSLOduration=20.7075063 podStartE2EDuration="20.7075063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.702982222 +0000 UTC m=+21.623686274" watchObservedRunningTime="2026-07-28 10:07:43.7075063 +0000 UTC m=+21.628210352"
Jul 28 10:07:43.721545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.721494    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l5m7h" podStartSLOduration=3.268965815 podStartE2EDuration="21.721478965s" totalImagesPullingTime="18.45251315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.721063011 +0000 UTC m=+21.641767063" watchObservedRunningTime="2026-07-28 10:07:43.721478965 +0000 UTC m=+21.642183016"
Jul 28 10:07:43.754563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.754521    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jt8td" podStartSLOduration=3.296245518 podStartE2EDuration="21.754506319s" totalImagesPullingTime="18.458260801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.738156598 +0000 UTC m=+21.658860650" watchObservedRunningTime="2026-07-28 10:07:43.754506319 +0000 UTC m=+21.675210373"
Jul 28 10:07:43.754949 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.754925    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hg6wf" podStartSLOduration=3.3692690499999998 podStartE2EDuration="21.75491909s" totalImagesPullingTime="18.38565004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.754279899 +0000 UTC m=+21.674983951" watchObservedRunningTime="2026-07-28 10:07:43.75491909 +0000 UTC m=+21.675623144"
Jul 28 10:07:43.774611 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.774559    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wdjkk" podStartSLOduration=3.293503749 podStartE2EDuration="21.774542422s" totalImagesPullingTime="18.481038673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.774000246 +0000 UTC m=+21.694704305" watchObservedRunningTime="2026-07-28 10:07:43.774542422 +0000 UTC m=+21.695246475"
Jul 28 10:07:43.791558 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:43.791511    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vf62c" podStartSLOduration=11.28388771 podStartE2EDuration="21.7914956s" totalImagesPullingTime="10.50760789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:43.791270622 +0000 UTC m=+21.711974698" watchObservedRunningTime="2026-07-28 10:07:43.7914956 +0000 UTC m=+21.712199663"
Jul 28 10:07:44.561443 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.561282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:44.561545 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:44.561526    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:44.612363 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.612340    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:44.613067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.613048    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:44.699464 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.699428    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="c8ed39e84d3927e393a7baf8ad6e4da2f0c7df1c18aa82bbe05c38f5bb01df2f" exitCode=0
Jul 28 10:07:44.700063 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.699502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"c8ed39e84d3927e393a7baf8ad6e4da2f0c7df1c18aa82bbe05c38f5bb01df2f"}
Jul 28 10:07:44.701979 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.701962    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:07:44.702249 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702232    2569 generic.go:356] "Generic (PLEG): container finished" podID="9a2d1abe-36a3-4e30-a01e-a238eba3decb" containerID="bf8a56db385bdf512fad920b4ecf78b507a9f79059d98ab09ff495ebab769abc" exitCode=1
Jul 28 10:07:44.702349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702329    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"4ea9bdb5e3053680b56e063521cead405e39181c51a5dbadde782ef68aafe6fc"}
Jul 28 10:07:44.702424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"c250c57915293030765fa87a24f7331854f07d612b8df427a930a867522477a8"}
Jul 28 10:07:44.702478 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"8560e783e68a48ef81e7d939d0d2be6ec0ef76d85565c2a50dc8b68b630d5ea5"}
Jul 28 10:07:44.702532 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"ffaf43ba9ad15c339a1210d244b4541d0b87f97cb4956db45450044bf3e8fe0a"}
Jul 28 10:07:44.702582 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerDied","Data":"bf8a56db385bdf512fad920b4ecf78b507a9f79059d98ab09ff495ebab769abc"}
Jul 28 10:07:44.702639 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.702622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"feb3f71803e9d993d9dbaddc68a1d0d0dafde98bde44f8db369ec7ea7deefa21"}
Jul 28 10:07:44.703061 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.703042    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:44.703258 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.703244    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vf62c"
Jul 28 10:07:44.914309 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:44.914289    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:07:45.459552 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.459443    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:07:44.914308173Z","UUID":"09fd8eab-f781-43c7-ba6a-5efa74141c60","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:07:45.461207 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.461185    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:07:45.461325 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.461232    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:07:45.560970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.560935    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:45.560970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.560960    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:45.561204 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:45.561076    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:45.561204 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:45.561181    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:45.706261 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.706218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-42b4h" event={"ID":"cddee4ee-e652-477b-838b-b04d06767bff","Type":"ContainerStarted","Data":"f4f8d134b3e153646612d9338f17ff30281828df5cdbf98b668af71d7ca483f1"}
Jul 28 10:07:45.708450 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.708415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx" event={"ID":"cb29c0a5-1710-4991-a3dc-65c32666afd5","Type":"ContainerStarted","Data":"0c63b2d990cea67bbf3eaf37ac7952c1e2085835dd73307d1cb42b3ed0f9341b"}
Jul 28 10:07:45.722653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:45.722554    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-42b4h" podStartSLOduration=5.271927911 podStartE2EDuration="23.722539251s" totalImagesPullingTime="18.45061134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:45.722011205 +0000 UTC m=+23.642715258" watchObservedRunningTime="2026-07-28 10:07:45.722539251 +0000 UTC m=+23.643243304"
Jul 28 10:07:46.560973 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:46.560939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:46.561156 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:46.561082    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:46.713586 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:46.713541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx" event={"ID":"cb29c0a5-1710-4991-a3dc-65c32666afd5","Type":"ContainerStarted","Data":"f2728de9462ba097692e42474745b8e12ee897ab547876f398be7fabcf4b56dd"}
Jul 28 10:07:46.736014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:46.735969    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfqdx" podStartSLOduration=4.079272063 podStartE2EDuration="24.735948132s" totalImagesPullingTime="20.656676069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:46.735246471 +0000 UTC m=+24.655950534" watchObservedRunningTime="2026-07-28 10:07:46.735948132 +0000 UTC m=+24.656652184"
Jul 28 10:07:47.561030 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:47.560828    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:47.561186 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:47.560829    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:47.561186 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:47.561118    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:47.561279 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:47.561220    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:47.719520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:47.719491    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:07:47.720076 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:47.719983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"dfc49da9984a6f48d0367fb9404ff245b7f1d87d55a05a5c7b3116d63a0595d7"}
Jul 28 10:07:48.560862 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:48.560830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:48.561052 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:48.560950    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:49.561606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.561409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:49.562204 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.561418    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:49.562204 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:49.561668    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:49.562204 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:49.561779    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:49.728797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.728731    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:07:49.729151 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.729116    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"82ddb11e8190503e553e75d9c64ba755e22f0f2b133e067708c98deac8d2b651"}
Jul 28 10:07:49.729373 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.729354    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:49.729526 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.729499    2569 scope.go:117] "RemoveContainer" containerID="bf8a56db385bdf512fad920b4ecf78b507a9f79059d98ab09ff495ebab769abc"
Jul 28 10:07:49.731036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.731016    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="a4f283e419c216c171b34129565d5e07d1905a72e6c9e409990ff055ce700ee1" exitCode=0
Jul 28 10:07:49.731133 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.731054    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"a4f283e419c216c171b34129565d5e07d1905a72e6c9e409990ff055ce700ee1"}
Jul 28 10:07:49.745515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:49.745488    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:50.561134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.561103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:50.561323 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:50.561232    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:50.740149 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.740127    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:07:50.740518 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.740491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" event={"ID":"9a2d1abe-36a3-4e30-a01e-a238eba3decb","Type":"ContainerStarted","Data":"822581a08efe836dbc28917b92939b5e22798b50ce373eb03ad765a6b79d5db0"}
Jul 28 10:07:50.740758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.740718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:50.740872 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.740805    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:50.755849 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.755827    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:07:50.776208 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:50.776165    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg" podStartSLOduration=9.887235846 podStartE2EDuration="28.776149652s" totalImagesPullingTime="18.888913806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:50.774994399 +0000 UTC m=+28.695698485" watchObservedRunningTime="2026-07-28 10:07:50.776149652 +0000 UTC m=+28.696853736"
Jul 28 10:07:51.166103 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.165841    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4rm74"]
Jul 28 10:07:51.166233 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.166165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:51.166295 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:51.166273    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:51.166496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.166473    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6k7q6"]
Jul 28 10:07:51.166626 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.166612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:51.166741 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:51.166719    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:51.167157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.167136    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2r6mj"]
Jul 28 10:07:51.167249 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.167232    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:51.167335 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:51.167318    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:51.598157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.598087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg6wf_395c1740-3932-4e87-bf52-06ee0acdd2b3/dns-node-resolver/0.log"
Jul 28 10:07:51.744659 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.744623    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="5e77e18460765f6a0f755d98b7c6b578c000f302d56dfc7c62d826d67247c6a6" exitCode=0
Jul 28 10:07:51.745028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:51.744714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"5e77e18460765f6a0f755d98b7c6b578c000f302d56dfc7c62d826d67247c6a6"}
Jul 28 10:07:52.562314 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:52.562280    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:52.562497 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:52.562398    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:52.562497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:52.562459    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:52.562627 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:52.562582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:52.575928 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:52.575898    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wdjkk_04758362-69bc-4f6f-a3e5-8ff5d2373d57/node-ca/0.log"
Jul 28 10:07:53.561390 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:53.561323    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:53.561944 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:53.561419    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:53.752395 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:53.752366    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="fb03a9b1af09d3795782daf0dcb88c1bdfb96cb13d53dbc453e86f7b67cf8bf3" exitCode=0
Jul 28 10:07:53.752543 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:53.752405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"fb03a9b1af09d3795782daf0dcb88c1bdfb96cb13d53dbc453e86f7b67cf8bf3"}
Jul 28 10:07:54.560746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:54.560709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:54.560933 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:54.560709    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:54.560933 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:54.560863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:54.561049 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:54.560989    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:55.561075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:55.561043    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:55.561562 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:55.561186    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:56.154303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:56.154267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:56.154322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:56.154356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154429    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154460    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154507    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154520    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret podName:92eeb473-d6b5-437e-a0dd-8a128f8b10d0 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:28.154499747 +0000 UTC m=+66.075203792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret") pod "global-pull-secret-syncer-6k7q6" (UID: "92eeb473-d6b5-437e-a0dd-8a128f8b10d0") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154528    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154541    2569 projected.go:196] Error preparing data for projected volume kube-api-access-gbhst for pod openshift-network-diagnostics/network-check-target-4rm74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:56.154604 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154597    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst podName:56b512b3-61eb-4eb5-9a48-1ea758c47797 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:28.154583202 +0000 UTC m=+66.075287234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gbhst" (UniqueName: "kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst") pod "network-check-target-4rm74" (UID: "56b512b3-61eb-4eb5-9a48-1ea758c47797") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:07:56.155104 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.154616    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs podName:46632d46-0a7f-4b86-99bd-65ad313076e8 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:28.154606331 +0000 UTC m=+66.075310367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs") pod "network-metrics-daemon-2r6mj" (UID: "46632d46-0a7f-4b86-99bd-65ad313076e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:07:56.561394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:56.561312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:56.561861 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:56.561312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:56.561861 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.561443    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:56.561861 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:56.561542    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:57.561063 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:57.561021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:57.561253 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:57.561157    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:58.560919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:58.560651    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:07:58.560919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:58.560715    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:07:58.561312 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:58.560932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:07:58.561312 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:58.560987    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:07:58.766801 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:58.766764    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="b967cea8acfae1986ab4434dc9b843e8d1331b9856f5feac907a4d0cffc5eb24" exitCode=0
Jul 28 10:07:58.766968 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:58.766825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"b967cea8acfae1986ab4434dc9b843e8d1331b9856f5feac907a4d0cffc5eb24"}
Jul 28 10:07:59.561671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:59.561635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:07:59.562057 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:07:59.561772    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:07:59.772258 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:59.772224    2569 generic.go:356] "Generic (PLEG): container finished" podID="1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612" containerID="39e9f060d9a3c095233002977df678541a315d4e535b91404a30a5bb98dd74b6" exitCode=0
Jul 28 10:07:59.772409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:07:59.772274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerDied","Data":"39e9f060d9a3c095233002977df678541a315d4e535b91404a30a5bb98dd74b6"}
Jul 28 10:08:00.561371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:00.561329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:00.561569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:00.561380    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:00.561569 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:00.561471    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:00.561569 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:00.561472    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:00.778190 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:00.778154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" event={"ID":"1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612","Type":"ContainerStarted","Data":"6658c629107951ae1cbaaa962316c7386a2c311c16eb711677bf5505c4285d2e"}
Jul 28 10:08:00.802831 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:00.802734    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dgh6v" podStartSLOduration=9.129905816 podStartE2EDuration="38.802719354s" totalImagesPullingTime="29.672813538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:00.802457677 +0000 UTC m=+38.723161740" watchObservedRunningTime="2026-07-28 10:08:00.802719354 +0000 UTC m=+38.723423406"
Jul 28 10:08:01.560783 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:01.560749    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:01.560951 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:01.560872    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:02.561466 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:02.561430    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:02.561971 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:02.561516    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:02.561971 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:02.561631    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:02.561971 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:02.561773    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:03.561510 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:03.561472    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:03.561923 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:03.561582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:04.561232 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:04.561192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:04.561469 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:04.561206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:04.561469 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:04.561335    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:04.561469 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:04.561387    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:05.561418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:05.561381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:05.561597 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:05.561489    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:06.561421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:06.561386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:06.561584 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:06.561391    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:06.561584 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:06.561496    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:06.561891 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:06.561637    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:07.561516 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:07.561482    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:07.561771 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:07.561588    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:08.560904 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:08.560869    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:08.561065 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:08.561005    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:08.561065 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:08.561052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:08.561157 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:08.561139    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:09.560989 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:09.560952    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:09.561361 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:09.561063    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:10.560933 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:10.560896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:10.561114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:10.560896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:10.561114 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:10.561003    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:10.561114 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:10.561076    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:11.561078 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:11.561044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:11.561260 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:11.561148    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:12.562078 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:12.562047    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:12.562433 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:12.562137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2r6mj" podUID="46632d46-0a7f-4b86-99bd-65ad313076e8"
Jul 28 10:08:12.562433 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:12.562164    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:12.562433 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:12.562206    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4rm74" podUID="56b512b3-61eb-4eb5-9a48-1ea758c47797"
Jul 28 10:08:13.561148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:13.561117    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:13.561317 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:13.561213    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k7q6" podUID="92eeb473-d6b5-437e-a0dd-8a128f8b10d0"
Jul 28 10:08:14.449894 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.449861    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-108.ec2.internal" event="NodeReady"
Jul 28 10:08:14.450322 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.449989    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:08:14.519175 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.519133    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xv85l"]
Jul 28 10:08:14.520804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.520784    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c6x8n"]
Jul 28 10:08:14.520939 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.520919    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.522144 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.522127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.525095 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.525077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6jd5\""
Jul 28 10:08:14.525430 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.525415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:08:14.525548 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.525535    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.525613 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.525574    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qspvm\""
Jul 28 10:08:14.525826 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.525809    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:08:14.526353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.526339    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.526460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.526441    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:08:14.540812 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.540786    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv85l"]
Jul 28 10:08:14.541537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.541519    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6x8n"]
Jul 28 10:08:14.554875 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.554845    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5z5"]
Jul 28 10:08:14.556661 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.556644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.560625 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.560605    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:14.560768 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.560755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:14.560768 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.560761    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:08:14.561511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.561493    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:08:14.561748 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.561724    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.562006 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.561989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tzznz\""
Jul 28 10:08:14.562561 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.562546    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.565362 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.565347    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:14.565667 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.565654    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c76md\""
Jul 28 10:08:14.565749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.565698    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.566434 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.566419    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.566538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.566523    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t2q9c\""
Jul 28 10:08:14.583389 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-data-volume\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.583468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.583468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzfz2\" (UniqueName: \"kubernetes.io/projected/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-api-access-jzfz2\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.583546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583475    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82fe5b6a-818c-4b00-9af5-0133aaebc62b-config-volume\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.583546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.583621 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d005d46-961d-4f0b-8fb1-a678331aa43f-cert\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.583621 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw4f\" (UniqueName: \"kubernetes.io/projected/5d005d46-961d-4f0b-8fb1-a678331aa43f-kube-api-access-2gw4f\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.583718 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kfh\" (UniqueName: \"kubernetes.io/projected/82fe5b6a-818c-4b00-9af5-0133aaebc62b-kube-api-access-w4kfh\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.583718 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82fe5b6a-818c-4b00-9af5-0133aaebc62b-metrics-tls\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.583797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-crio-socket\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.583797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.583778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fe5b6a-818c-4b00-9af5-0133aaebc62b-tmp-dir\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.585744 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.585722    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5z5"]
Jul 28 10:08:14.684287 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82fe5b6a-818c-4b00-9af5-0133aaebc62b-metrics-tls\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.684287 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-crio-socket\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fe5b6a-818c-4b00-9af5-0133aaebc62b-tmp-dir\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.684519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-data-volume\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684508    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-crio-socket\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzfz2\" (UniqueName: \"kubernetes.io/projected/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-api-access-jzfz2\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82fe5b6a-818c-4b00-9af5-0133aaebc62b-config-volume\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.684746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684638    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82fe5b6a-818c-4b00-9af5-0133aaebc62b-tmp-dir\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.684746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.684946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d005d46-961d-4f0b-8fb1-a678331aa43f-cert\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.684946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw4f\" (UniqueName: \"kubernetes.io/projected/5d005d46-961d-4f0b-8fb1-a678331aa43f-kube-api-access-2gw4f\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.684946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kfh\" (UniqueName: \"kubernetes.io/projected/82fe5b6a-818c-4b00-9af5-0133aaebc62b-kube-api-access-w4kfh\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.684946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.684855    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-data-volume\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.685172 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.685148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.685240 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.685153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82fe5b6a-818c-4b00-9af5-0133aaebc62b-config-volume\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.688366 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.688338    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.688499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.688344    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82fe5b6a-818c-4b00-9af5-0133aaebc62b-metrics-tls\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.688499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.688432    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d005d46-961d-4f0b-8fb1-a678331aa43f-cert\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.699606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.699584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kfh\" (UniqueName: \"kubernetes.io/projected/82fe5b6a-818c-4b00-9af5-0133aaebc62b-kube-api-access-w4kfh\") pod \"dns-default-xv85l\" (UID: \"82fe5b6a-818c-4b00-9af5-0133aaebc62b\") " pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.699713 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.699657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw4f\" (UniqueName: \"kubernetes.io/projected/5d005d46-961d-4f0b-8fb1-a678331aa43f-kube-api-access-2gw4f\") pod \"ingress-canary-c6x8n\" (UID: \"5d005d46-961d-4f0b-8fb1-a678331aa43f\") " pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.700023 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.699988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzfz2\" (UniqueName: \"kubernetes.io/projected/5eb49b5a-7beb-4d29-9432-ff5a3c35de8b-kube-api-access-jzfz2\") pod \"insights-runtime-extractor-fr5z5\" (UID: \"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b\") " pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.831519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.831487    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:14.836007 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.835989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c6x8n"
Jul 28 10:08:14.866843 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.866812    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fr5z5"
Jul 28 10:08:14.986367 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:14.984879    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c6x8n"]
Jul 28 10:08:15.009157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.009130    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xv85l"]
Jul 28 10:08:15.012023 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:15.011993    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fe5b6a_818c_4b00_9af5_0133aaebc62b.slice/crio-c53c34dd7572fd1adbeec9057b231ae4294a2e980faf383cd7d3281687aa2524 WatchSource:0}: Error finding container c53c34dd7572fd1adbeec9057b231ae4294a2e980faf383cd7d3281687aa2524: Status 404 returned error can't find the container with id c53c34dd7572fd1adbeec9057b231ae4294a2e980faf383cd7d3281687aa2524
Jul 28 10:08:15.016871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.016786    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fr5z5"]
Jul 28 10:08:15.020909 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:15.020886    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb49b5a_7beb_4d29_9432_ff5a3c35de8b.slice/crio-eaa9ccc8766b6d1d52d1636a686660854de1d2b65c5c3743d13f3ebe834610ec WatchSource:0}: Error finding container eaa9ccc8766b6d1d52d1636a686660854de1d2b65c5c3743d13f3ebe834610ec: Status 404 returned error can't find the container with id eaa9ccc8766b6d1d52d1636a686660854de1d2b65c5c3743d13f3ebe834610ec
Jul 28 10:08:15.561523 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.561488    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:15.564785 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.564763    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:15.813842 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.813698    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5z5" event={"ID":"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b","Type":"ContainerStarted","Data":"eaa9ccc8766b6d1d52d1636a686660854de1d2b65c5c3743d13f3ebe834610ec"}
Jul 28 10:08:15.815083 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.815026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv85l" event={"ID":"82fe5b6a-818c-4b00-9af5-0133aaebc62b","Type":"ContainerStarted","Data":"c53c34dd7572fd1adbeec9057b231ae4294a2e980faf383cd7d3281687aa2524"}
Jul 28 10:08:15.816355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:15.816302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6x8n" event={"ID":"5d005d46-961d-4f0b-8fb1-a678331aa43f","Type":"ContainerStarted","Data":"6065c479b0b024471447b6d8d328d2defec980b655459c915ccb240ed6a5a318"}
Jul 28 10:08:21.834895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.834537    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv85l" event={"ID":"82fe5b6a-818c-4b00-9af5-0133aaebc62b","Type":"ContainerStarted","Data":"bc3ee8b6d6d09870cc3bd01517339056e634ac22471dbed1c4b1aa6f08dfa64c"}
Jul 28 10:08:21.834895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.834655    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:21.834895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.834698    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xv85l" event={"ID":"82fe5b6a-818c-4b00-9af5-0133aaebc62b","Type":"ContainerStarted","Data":"9184742a8df7a5e66866689fe90da32568d2894cd3dc54a077bbc9afd65ac667"}
Jul 28 10:08:21.836326 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.836296    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c6x8n" event={"ID":"5d005d46-961d-4f0b-8fb1-a678331aa43f","Type":"ContainerStarted","Data":"80fe9664b1a1ea646b2bfa75766cdefd255b6425950df18f2308696fc5dc3359"}
Jul 28 10:08:21.837754 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.837728    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5z5" event={"ID":"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b","Type":"ContainerStarted","Data":"a039306f32c3f9255f7efe908803d83fc7433dd2e761ca8cf72fa6844bb86553"}
Jul 28 10:08:21.857227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.857166    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xv85l" podStartSLOduration=2.143532863 podStartE2EDuration="7.857145463s" totalImagesPullingTime="5.7136126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:21.855056463 +0000 UTC m=+59.775760528" watchObservedRunningTime="2026-07-28 10:08:21.857145463 +0000 UTC m=+59.777849496"
Jul 28 10:08:21.876838 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:21.876791    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c6x8n" podStartSLOduration=2.137904866 podStartE2EDuration="7.876774657s" totalImagesPullingTime="5.738869791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:21.876462402 +0000 UTC m=+59.797166456" watchObservedRunningTime="2026-07-28 10:08:21.876774657 +0000 UTC m=+59.797478708"
Jul 28 10:08:22.767347 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:22.767315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kvnkg"
Jul 28 10:08:24.852374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:24.852333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5z5" event={"ID":"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b","Type":"ContainerStarted","Data":"50c9fca2e61dd9c0679affc8837dbe8d10022517358d3d0e505367395380aaaa"}
Jul 28 10:08:25.562414 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.562372    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"]
Jul 28 10:08:25.564351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.564317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:25.567563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.567539    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 10:08:25.568010 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.567954    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gkpfd\""
Jul 28 10:08:25.585181 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.585147    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"]
Jul 28 10:08:25.673959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.673927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mlgl5\" (UID: \"94652a70-76f8-40cb-b4b6-cc7030a21069\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:25.774353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:25.774319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mlgl5\" (UID: \"94652a70-76f8-40cb-b4b6-cc7030a21069\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:25.774542 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:25.774498    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 10:08:25.774593 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:25.774569    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates podName:94652a70-76f8-40cb-b4b6-cc7030a21069 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:26.274551604 +0000 UTC m=+64.195255633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-mlgl5" (UID: "94652a70-76f8-40cb-b4b6-cc7030a21069") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 10:08:26.279619 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:26.279583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mlgl5\" (UID: \"94652a70-76f8-40cb-b4b6-cc7030a21069\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:26.282229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:26.282203    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/94652a70-76f8-40cb-b4b6-cc7030a21069-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-mlgl5\" (UID: \"94652a70-76f8-40cb-b4b6-cc7030a21069\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:26.475535 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:26.475493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:27.857194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:27.857163    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"]
Jul 28 10:08:27.861985 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:27.861955    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94652a70_76f8_40cb_b4b6_cc7030a21069.slice/crio-c2d46d938ff4e40b567f944732517b746074339dcb773e243ac4bb64beb637df WatchSource:0}: Error finding container c2d46d938ff4e40b567f944732517b746074339dcb773e243ac4bb64beb637df: Status 404 returned error can't find the container with id c2d46d938ff4e40b567f944732517b746074339dcb773e243ac4bb64beb637df
Jul 28 10:08:27.864133 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:27.864035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fr5z5" event={"ID":"5eb49b5a-7beb-4d29-9432-ff5a3c35de8b","Type":"ContainerStarted","Data":"9636c24e7500fec32661d20f06be931935e8dd5542f79cad869d925f536954f1"}
Jul 28 10:08:28.193391 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.193340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:28.193587 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.193402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:28.193587 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.193438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:28.196513 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.196494    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:28.197075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.197062    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:28.197510 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.197495    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:28.206583 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.206561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92eeb473-d6b5-437e-a0dd-8a128f8b10d0-original-pull-secret\") pod \"global-pull-secret-syncer-6k7q6\" (UID: \"92eeb473-d6b5-437e-a0dd-8a128f8b10d0\") " pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:28.206755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.206738    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46632d46-0a7f-4b86-99bd-65ad313076e8-metrics-certs\") pod \"network-metrics-daemon-2r6mj\" (UID: \"46632d46-0a7f-4b86-99bd-65ad313076e8\") " pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:28.207420 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.207404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:28.216847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.216826    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhst\" (UniqueName: \"kubernetes.io/projected/56b512b3-61eb-4eb5-9a48-1ea758c47797-kube-api-access-gbhst\") pod \"network-check-target-4rm74\" (UID: \"56b512b3-61eb-4eb5-9a48-1ea758c47797\") " pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:28.375513 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.375481    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t2q9c\""
Jul 28 10:08:28.379016 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.378989    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c76md\""
Jul 28 10:08:28.383243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.383223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:28.387024 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.386991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2r6mj"
Jul 28 10:08:28.473285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.473036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k7q6"
Jul 28 10:08:28.529482 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.529248    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fr5z5" podStartSLOduration=1.92071349 podStartE2EDuration="14.529226891s" totalImagesPullingTime="12.608513401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:27.888562673 +0000 UTC m=+65.809266725" watchObservedRunningTime="2026-07-28 10:08:28.529226891 +0000 UTC m=+66.449930946"
Jul 28 10:08:28.530443 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.530417    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4rm74"]
Jul 28 10:08:28.539064 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:28.539016    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b512b3_61eb_4eb5_9a48_1ea758c47797.slice/crio-4e26b0241a4448cf7462bebbe2524b9a26db67619b8d7b2135a8494a5b548c48 WatchSource:0}: Error finding container 4e26b0241a4448cf7462bebbe2524b9a26db67619b8d7b2135a8494a5b548c48: Status 404 returned error can't find the container with id 4e26b0241a4448cf7462bebbe2524b9a26db67619b8d7b2135a8494a5b548c48
Jul 28 10:08:28.545861 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.545622    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2r6mj"]
Jul 28 10:08:28.549644 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:28.549604    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46632d46_0a7f_4b86_99bd_65ad313076e8.slice/crio-770d4736c2b58bd98bbe5bd24e0ec2497830d28439e9b257c35ce3d38d26c9ff WatchSource:0}: Error finding container 770d4736c2b58bd98bbe5bd24e0ec2497830d28439e9b257c35ce3d38d26c9ff: Status 404 returned error can't find the container with id 770d4736c2b58bd98bbe5bd24e0ec2497830d28439e9b257c35ce3d38d26c9ff
Jul 28 10:08:28.623874 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.623836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6k7q6"]
Jul 28 10:08:28.627744 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:28.627715    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92eeb473_d6b5_437e_a0dd_8a128f8b10d0.slice/crio-25399040703d28f569788b5f32dec25cfb5595553bcccc04dfd821b8a77da9e2 WatchSource:0}: Error finding container 25399040703d28f569788b5f32dec25cfb5595553bcccc04dfd821b8a77da9e2: Status 404 returned error can't find the container with id 25399040703d28f569788b5f32dec25cfb5595553bcccc04dfd821b8a77da9e2
Jul 28 10:08:28.871127 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.871028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5" event={"ID":"94652a70-76f8-40cb-b4b6-cc7030a21069","Type":"ContainerStarted","Data":"c2d46d938ff4e40b567f944732517b746074339dcb773e243ac4bb64beb637df"}
Jul 28 10:08:28.872393 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.872358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r6mj" event={"ID":"46632d46-0a7f-4b86-99bd-65ad313076e8","Type":"ContainerStarted","Data":"770d4736c2b58bd98bbe5bd24e0ec2497830d28439e9b257c35ce3d38d26c9ff"}
Jul 28 10:08:28.873704 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.873662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4rm74" event={"ID":"56b512b3-61eb-4eb5-9a48-1ea758c47797","Type":"ContainerStarted","Data":"4e26b0241a4448cf7462bebbe2524b9a26db67619b8d7b2135a8494a5b548c48"}
Jul 28 10:08:28.874985 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:28.874960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6k7q6" event={"ID":"92eeb473-d6b5-437e-a0dd-8a128f8b10d0","Type":"ContainerStarted","Data":"25399040703d28f569788b5f32dec25cfb5595553bcccc04dfd821b8a77da9e2"}
Jul 28 10:08:29.881542 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:29.881507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5" event={"ID":"94652a70-76f8-40cb-b4b6-cc7030a21069","Type":"ContainerStarted","Data":"87f42fb92ce9f1ab3c23c6feb930e93d702b44a115ee1eb56dd1d4d24a3622c8"}
Jul 28 10:08:29.881955 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:29.881722    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:29.887003 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:29.886978    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5"
Jul 28 10:08:29.908606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:29.908548    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-mlgl5" podStartSLOduration=3.803813773 podStartE2EDuration="4.908530056s" totalImagesPullingTime="1.104716283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:29.908018875 +0000 UTC m=+67.828722925" watchObservedRunningTime="2026-07-28 10:08:29.908530056 +0000 UTC m=+67.829234107"
Jul 28 10:08:30.630210 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.630174    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"]
Jul 28 10:08:30.632355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.632325    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.635056 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.635030    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t6dql\""
Jul 28 10:08:30.636150 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.636123    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:08:30.636265 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.636179    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:08:30.636424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.636408    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 10:08:30.636509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.636491    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:08:30.636610 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.636586    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 10:08:30.645323 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.645270    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"]
Jul 28 10:08:30.709050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.708956    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.709050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.709028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.709286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.709069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.709286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.709110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cs99\" (UniqueName: \"kubernetes.io/projected/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-kube-api-access-8cs99\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.810735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.810440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.810735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.810507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs99\" (UniqueName: \"kubernetes.io/projected/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-kube-api-access-8cs99\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.810735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.810558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.810735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.810599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.811769 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.811736    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.813489 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.813461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.814179 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.814143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.820537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.820496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs99\" (UniqueName: \"kubernetes.io/projected/0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e-kube-api-access-8cs99\") pod \"prometheus-operator-84d69f794d-lxmx4\" (UID: \"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:30.887801 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.887760    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r6mj" event={"ID":"46632d46-0a7f-4b86-99bd-65ad313076e8","Type":"ContainerStarted","Data":"cb203862eb2f99c13399ebd8cdc0e3ef7030f764789373cfd9e20896073506e4"}
Jul 28 10:08:30.888255 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.887823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2r6mj" event={"ID":"46632d46-0a7f-4b86-99bd-65ad313076e8","Type":"ContainerStarted","Data":"681c2551e8ece8c0c7912586335d144272178648fb504cba5283f27dac8ed553"}
Jul 28 10:08:30.907397 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.907321    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2r6mj" podStartSLOduration=67.272461414 podStartE2EDuration="1m8.907300772s" totalImagesPullingTime="1.634839358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:30.906217838 +0000 UTC m=+68.826921889" watchObservedRunningTime="2026-07-28 10:08:30.907300772 +0000 UTC m=+68.828004821"
Jul 28 10:08:30.943243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:30.943206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"
Jul 28 10:08:31.652472 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.652432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lxmx4"]
Jul 28 10:08:31.657268 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:31.657234    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1d5989_1e2d_4c6a_aaec_be4e767d8e4e.slice/crio-cbbcc846d4ff471f1afb1d3ac6673200d0ccfdf0b9e2ea62e85ff938fc3a9b0a WatchSource:0}: Error finding container cbbcc846d4ff471f1afb1d3ac6673200d0ccfdf0b9e2ea62e85ff938fc3a9b0a: Status 404 returned error can't find the container with id cbbcc846d4ff471f1afb1d3ac6673200d0ccfdf0b9e2ea62e85ff938fc3a9b0a
Jul 28 10:08:31.843540 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.843463    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xv85l"
Jul 28 10:08:31.893626 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.893591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4rm74" event={"ID":"56b512b3-61eb-4eb5-9a48-1ea758c47797","Type":"ContainerStarted","Data":"fa46bd51bcb18604047e9606345af4fca47ac173f829dbbb4671001df1d738d8"}
Jul 28 10:08:31.894103 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.893708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:08:31.895236 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.895158    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4" event={"ID":"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e","Type":"ContainerStarted","Data":"cbbcc846d4ff471f1afb1d3ac6673200d0ccfdf0b9e2ea62e85ff938fc3a9b0a"}
Jul 28 10:08:31.914214 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:31.914159    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4rm74" podStartSLOduration=66.895379968 podStartE2EDuration="1m9.914140187s" totalImagesPullingTime="3.018760219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:31.912920176 +0000 UTC m=+69.833624225" watchObservedRunningTime="2026-07-28 10:08:31.914140187 +0000 UTC m=+69.834844239"
Jul 28 10:08:34.905833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:34.905798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6k7q6" event={"ID":"92eeb473-d6b5-437e-a0dd-8a128f8b10d0","Type":"ContainerStarted","Data":"286dc0ea5238442a993e250c5562cebda4503908041505ad45bb61f5f2581ebb"}
Jul 28 10:08:34.907521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:34.907498    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4" event={"ID":"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e","Type":"ContainerStarted","Data":"f76b55e91283210e3bb8ae25311daca467a82c8ea9f62294ee7f5ad0fb17f326"}
Jul 28 10:08:34.907637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:34.907545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4" event={"ID":"0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e","Type":"ContainerStarted","Data":"09eba8607970457216ca6794a0c7218da963e340f181062ab250e8a3e3ba0cfe"}
Jul 28 10:08:34.928497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:34.928448    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6k7q6" podStartSLOduration=67.61112752 podStartE2EDuration="1m12.92843352s" totalImagesPullingTime="5.317306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.928002658 +0000 UTC m=+72.848706710" watchObservedRunningTime="2026-07-28 10:08:34.92843352 +0000 UTC m=+72.849137573"
Jul 28 10:08:34.947403 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:34.947328    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-lxmx4" podStartSLOduration=2.664422556 podStartE2EDuration="4.947314267s" totalImagesPullingTime="2.282891711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.945909703 +0000 UTC m=+72.866613756" watchObservedRunningTime="2026-07-28 10:08:34.947314267 +0000 UTC m=+72.868018333"
Jul 28 10:08:35.809695 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.809652    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-5vpqg"]
Jul 28 10:08:35.812630 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.812614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:35.816484 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.816461    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 10:08:35.816586 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.816510    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fgvcv\""
Jul 28 10:08:35.816850 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.816834    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 10:08:35.827787 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.827763    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-5vpqg"]
Jul 28 10:08:35.948238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:35.948197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8tb\" (UniqueName: \"kubernetes.io/projected/5fef08c7-1084-4238-b972-baadace7894b-kube-api-access-vk8tb\") pod \"downloads-7c97dc4555-5vpqg\" (UID: \"5fef08c7-1084-4238-b972-baadace7894b\") " pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:36.049495 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:36.049456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8tb\" (UniqueName: \"kubernetes.io/projected/5fef08c7-1084-4238-b972-baadace7894b-kube-api-access-vk8tb\") pod \"downloads-7c97dc4555-5vpqg\" (UID: \"5fef08c7-1084-4238-b972-baadace7894b\") " pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:36.060590 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:36.060530    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8tb\" (UniqueName: \"kubernetes.io/projected/5fef08c7-1084-4238-b972-baadace7894b-kube-api-access-vk8tb\") pod \"downloads-7c97dc4555-5vpqg\" (UID: \"5fef08c7-1084-4238-b972-baadace7894b\") " pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:36.121377 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:36.121344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:36.250527 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:36.250493    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-5vpqg"]
Jul 28 10:08:36.254613 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:36.254581    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fef08c7_1084_4238_b972_baadace7894b.slice/crio-365db0496ed6c48d12b239ff1720441a3d3461548ea8aaffb62a00ca303e6827 WatchSource:0}: Error finding container 365db0496ed6c48d12b239ff1720441a3d3461548ea8aaffb62a00ca303e6827: Status 404 returned error can't find the container with id 365db0496ed6c48d12b239ff1720441a3d3461548ea8aaffb62a00ca303e6827
Jul 28 10:08:36.915017 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:36.914980    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-5vpqg" event={"ID":"5fef08c7-1084-4238-b972-baadace7894b","Type":"ContainerStarted","Data":"365db0496ed6c48d12b239ff1720441a3d3461548ea8aaffb62a00ca303e6827"}
Jul 28 10:08:37.037917 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.037880    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w5hlp"]
Jul 28 10:08:37.042216 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.042198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.045841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.045817    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:08:37.045841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.045828    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:08:37.046036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.045821    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r255l\""
Jul 28 10:08:37.046036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.045895    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:08:37.156835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.156806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqbj\" (UniqueName: \"kubernetes.io/projected/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-kube-api-access-xhqbj\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.156849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.156872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-wtmp\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.156896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.156977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-sys\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.157013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-textfile\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.157034    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-metrics-client-ca\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.157094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-root\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.157241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.157130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-tls\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258459    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqbj\" (UniqueName: \"kubernetes.io/projected/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-kube-api-access-xhqbj\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258554    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-wtmp\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258576    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-sys\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-textfile\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-metrics-client-ca\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-root\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.258824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-tls\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.258911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-sys\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.259054    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-root\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259284 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.259266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-wtmp\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.259441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-textfile\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259583 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.259561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-accelerators-collector-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.259672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.259654    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-metrics-client-ca\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.261164 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.261144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-tls\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.261579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.261563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.267420 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.267400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqbj\" (UniqueName: \"kubernetes.io/projected/a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9-kube-api-access-xhqbj\") pod \"node-exporter-w5hlp\" (UID: \"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9\") " pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.351545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.351501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w5hlp"
Jul 28 10:08:37.921571 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:37.921483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5hlp" event={"ID":"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9","Type":"ContainerStarted","Data":"901a5cb5c83da87c10c6d702277eaedbbc15ea4ffe8ceba4b1a1c382cf97f91d"}
Jul 28 10:08:38.927229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:38.927191    2569 generic.go:356] "Generic (PLEG): container finished" podID="a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9" containerID="e83dd9b776ca9ead3e11c004f5adc7cbe5f90e124794cee80d181af9117df81b" exitCode=0
Jul 28 10:08:38.927726 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:38.927275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5hlp" event={"ID":"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9","Type":"ContainerDied","Data":"e83dd9b776ca9ead3e11c004f5adc7cbe5f90e124794cee80d181af9117df81b"}
Jul 28 10:08:39.064480 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.064456    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8c959989b-b5krg"]
Jul 28 10:08:39.068145 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.068095    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.072974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.072927    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-15aav9lkkbvb8\""
Jul 28 10:08:39.072974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.072957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l9mf6\""
Jul 28 10:08:39.073745 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.073513    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 10:08:39.073745 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.073527    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 10:08:39.073745 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.073554    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 10:08:39.073955 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.073762    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 10:08:39.073955 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.073853    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 10:08:39.090023 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.089989    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8c959989b-b5krg"]
Jul 28 10:08:39.173066 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173033    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003e27be-6245-47da-87d4-49ed86acf3e7-metrics-client-ca\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173066 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxg2\" (UniqueName: \"kubernetes.io/projected/003e27be-6245-47da-87d4-49ed86acf3e7-kube-api-access-mkxg2\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173267    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-grpc-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173467 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173322    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.173467 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.173356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-grpc-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274052    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274365 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274365 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003e27be-6245-47da-87d4-49ed86acf3e7-metrics-client-ca\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274365 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxg2\" (UniqueName: \"kubernetes.io/projected/003e27be-6245-47da-87d4-49ed86acf3e7-kube-api-access-mkxg2\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274365 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.274365 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.274231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.275267 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.275233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003e27be-6245-47da-87d4-49ed86acf3e7-metrics-client-ca\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277207 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-grpc-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277851    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-tls\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.277990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.277927    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/003e27be-6245-47da-87d4-49ed86acf3e7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.286402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.286377    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxg2\" (UniqueName: \"kubernetes.io/projected/003e27be-6245-47da-87d4-49ed86acf3e7-kube-api-access-mkxg2\") pod \"thanos-querier-8c959989b-b5krg\" (UID: \"003e27be-6245-47da-87d4-49ed86acf3e7\") " pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.379833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.379796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:39.534728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.534539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8c959989b-b5krg"]
Jul 28 10:08:39.538427 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:39.538398    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003e27be_6245_47da_87d4_49ed86acf3e7.slice/crio-c2ad857db385a84ed66a05ab09200207078645ade26c3e6c277c54591e870ca7 WatchSource:0}: Error finding container c2ad857db385a84ed66a05ab09200207078645ade26c3e6c277c54591e870ca7: Status 404 returned error can't find the container with id c2ad857db385a84ed66a05ab09200207078645ade26c3e6c277c54591e870ca7
Jul 28 10:08:39.934034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.934000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5hlp" event={"ID":"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9","Type":"ContainerStarted","Data":"82d25c7ca8cd61b287825318f86dd08ba5fb21c057201e4ee4a73445eed7e557"}
Jul 28 10:08:39.934497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.934095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w5hlp" event={"ID":"a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9","Type":"ContainerStarted","Data":"62de81fe451da5d4539c99705ee4ea6a94022e6582c41fcde068cb3b6d1156de"}
Jul 28 10:08:39.935284 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.935255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"c2ad857db385a84ed66a05ab09200207078645ade26c3e6c277c54591e870ca7"}
Jul 28 10:08:39.961590 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:39.961529    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w5hlp" podStartSLOduration=1.718633055 podStartE2EDuration="2.961506763s" totalImagesPullingTime="1.169770674s" totalInitContainerRuntime="73.103034ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:39.960810268 +0000 UTC m=+77.881514319" watchObservedRunningTime="2026-07-28 10:08:39.961506763 +0000 UTC m=+77.882210815"
Jul 28 10:08:41.386472 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.386439    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-847755dcc8-kl6ll"]
Jul 28 10:08:41.389788 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.389760    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.392491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.392467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 10:08:41.392630 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.392540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 10:08:41.393789 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.393754    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 10:08:41.393964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.393950    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6nh8ufkfe5h0m\""
Jul 28 10:08:41.394563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.394541    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-npjkv\""
Jul 28 10:08:41.396433 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.394944    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 10:08:41.403742 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.403145    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847755dcc8-kl6ll"]
Jul 28 10:08:41.496855 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-client-certs\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.496855 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g76fn\" (UniqueName: \"kubernetes.io/projected/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-kube-api-access-g76fn\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.497081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.497081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496946    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-tls\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.497081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-client-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.497081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.496987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-metrics-server-audit-profiles\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.497081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.497026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-audit-log\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.597923 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.597883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.597923 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.597930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-tls\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.597948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-client-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.597970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-metrics-server-audit-profiles\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.598011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-audit-log\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.598048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-client-certs\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.598070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g76fn\" (UniqueName: \"kubernetes.io/projected/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-kube-api-access-g76fn\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598727 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.598667    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-audit-log\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.598811 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.598753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.599370 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.599344    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-metrics-server-audit-profiles\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.601276 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.601256    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-client-certs\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.601392 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.601356    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-client-ca-bundle\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.601456 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.601418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-secret-metrics-server-tls\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.607035 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.607011    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g76fn\" (UniqueName: \"kubernetes.io/projected/b2b4cb46-8f76-4f9b-b140-3d7af40dd596-kube-api-access-g76fn\") pod \"metrics-server-847755dcc8-kl6ll\" (UID: \"b2b4cb46-8f76-4f9b-b140-3d7af40dd596\") " pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.704977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.704946    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:08:41.741042 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.740977    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"]
Jul 28 10:08:41.744427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.744402    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:41.747143 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.747115    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jdmp4\""
Jul 28 10:08:41.747262 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.747138    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 10:08:41.757080 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.757009    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"]
Jul 28 10:08:41.864132 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.864101    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-847755dcc8-kl6ll"]
Jul 28 10:08:41.900499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.900372    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98fg\" (UniqueName: \"kubernetes.io/projected/8683564a-8905-45ba-8bd3-ca7b5531e249-kube-api-access-b98fg\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:41.900499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:41.900436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.001178 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.001095    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98fg\" (UniqueName: \"kubernetes.io/projected/8683564a-8905-45ba-8bd3-ca7b5531e249-kube-api-access-b98fg\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.001178 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.001151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.001395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:42.001270    2569 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 10:08:42.001395 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:08:42.001360    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert podName:8683564a-8905-45ba-8bd3-ca7b5531e249 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:42.501335247 +0000 UTC m=+80.422039277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-z5rmx" (UID: "8683564a-8905-45ba-8bd3-ca7b5531e249") : secret "monitoring-plugin-cert" not found
Jul 28 10:08:42.010589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.010532    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98fg\" (UniqueName: \"kubernetes.io/projected/8683564a-8905-45ba-8bd3-ca7b5531e249-kube-api-access-b98fg\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.373334 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:42.373258    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b4cb46_8f76_4f9b_b140_3d7af40dd596.slice/crio-1151d62f12c13bb9ca27c594e150dd517fb4932793897c6d1c05641ebb5d100a WatchSource:0}: Error finding container 1151d62f12c13bb9ca27c594e150dd517fb4932793897c6d1c05641ebb5d100a: Status 404 returned error can't find the container with id 1151d62f12c13bb9ca27c594e150dd517fb4932793897c6d1c05641ebb5d100a
Jul 28 10:08:42.505790 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.505765    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.508212 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.508193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8683564a-8905-45ba-8bd3-ca7b5531e249-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z5rmx\" (UID: \"8683564a-8905-45ba-8bd3-ca7b5531e249\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.657620 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.657512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:42.797817 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.797782    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"]
Jul 28 10:08:42.801283 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:08:42.801244    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8683564a_8905_45ba_8bd3_ca7b5531e249.slice/crio-794b7a295f72792b3c7429fdfb620c7a5c4961206c567464d0a1f6e40d9269ef WatchSource:0}: Error finding container 794b7a295f72792b3c7429fdfb620c7a5c4961206c567464d0a1f6e40d9269ef: Status 404 returned error can't find the container with id 794b7a295f72792b3c7429fdfb620c7a5c4961206c567464d0a1f6e40d9269ef
Jul 28 10:08:42.949312 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.949273    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx" event={"ID":"8683564a-8905-45ba-8bd3-ca7b5531e249","Type":"ContainerStarted","Data":"794b7a295f72792b3c7429fdfb620c7a5c4961206c567464d0a1f6e40d9269ef"}
Jul 28 10:08:42.951670 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.951640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"2ff422c9242b30af2be00fdf8909a75be057ffefd2a4fc27444a9ea27adf2bb5"}
Jul 28 10:08:42.951835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.951734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"7d083bbfc8a6bb5f8d6a12c5424515a439f80a3d376bb8d7b38e8645069dd4ef"}
Jul 28 10:08:42.951835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.951779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"6a075e7be90e1e3b42f4fb68be5ca4f43b65c270087f51ccfa72b30a634e4d82"}
Jul 28 10:08:42.952924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:42.952901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll" event={"ID":"b2b4cb46-8f76-4f9b-b140-3d7af40dd596","Type":"ContainerStarted","Data":"1151d62f12c13bb9ca27c594e150dd517fb4932793897c6d1c05641ebb5d100a"}
Jul 28 10:08:43.323581 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.323330    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:08:43.326546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.326519    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.332624 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.332566    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 10:08:43.332624 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.332619    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 10:08:43.334731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.333667    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 10:08:43.334731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.333849    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 10:08:43.334731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.333967    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 10:08:43.334731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.333991    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 10:08:43.334731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.334599    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xzkds\""
Jul 28 10:08:43.335043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.334750    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 10:08:43.335614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.335213    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 10:08:43.335614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.335274    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 10:08:43.335614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.335215    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 10:08:43.335835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.335745    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 10:08:43.341889 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.337453    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 10:08:43.341889 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.338013    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ds848bs3kce69\""
Jul 28 10:08:43.345231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.345207    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 10:08:43.347920 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.347883    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 10:08:43.415255 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415440 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415340    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415519 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415663    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415733    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.415749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415799    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5v9\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415831    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.416046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.415972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.429017 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.428877    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5v9\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518491    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518526    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.518729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518588    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.519951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.519951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518626    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.519951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.518641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.520863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.521366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.522024    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522505 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.522316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522808 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.522573    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.522808 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.522678    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.524505 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.524187    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.526863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.526514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.527797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.527775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.527921 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.527884    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.530366 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.530343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.530588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.530600    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.530868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.531055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.531075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.531388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.531326    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.533579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.533527    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.534337 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.534316    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.537245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.537224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5v9\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9\") pod \"prometheus-k8s-0\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.646014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.645680    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:08:43.926703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.926656    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:08:43.962809 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.962774    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"08acb822db0373ac2f2524d64fcf6e060d545078206ae64ab628f164332e1645"}
Jul 28 10:08:43.962910 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.962861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"5cc9a6a91d586e575067823b1c4f1f0b78dfc3bdbc0efa53549b4c749bb95bb2"}
Jul 28 10:08:43.964479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:43.964454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"3f304f53fa9f3850f9edd4eb72dbad64857e20fa35fc1cf485baca353e071377"}
Jul 28 10:08:44.978778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:44.978730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" event={"ID":"003e27be-6245-47da-87d4-49ed86acf3e7","Type":"ContainerStarted","Data":"259a0d14ed6b178e9dc35611cf7d9973bac5a560705f40a9d45671b69d3ba1de"}
Jul 28 10:08:44.979196 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:44.978960    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:45.006945 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.006876    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg" podStartSLOduration=2.040200159 podStartE2EDuration="6.006852486s" totalImagesPullingTime="3.966652327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:45.005871297 +0000 UTC m=+82.926575350" watchObservedRunningTime="2026-07-28 10:08:45.006852486 +0000 UTC m=+82.927556538"
Jul 28 10:08:45.986758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.986610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx" event={"ID":"8683564a-8905-45ba-8bd3-ca7b5531e249","Type":"ContainerStarted","Data":"8c5a8912c2616f53ef8edb1edb06a774b3725d5e7f5c6c1c088771e70aadbd6e"}
Jul 28 10:08:45.986758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.986702    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:45.988662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.988422    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll" event={"ID":"b2b4cb46-8f76-4f9b-b140-3d7af40dd596","Type":"ContainerStarted","Data":"82efad22ae1e27615c813c8a818452d66b3ec4be2db665a15a633599acf440e6"}
Jul 28 10:08:45.990870 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.990761    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e" exitCode=0
Jul 28 10:08:45.991030 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.990902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"}
Jul 28 10:08:45.993501 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:45.993458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx"
Jul 28 10:08:46.011332 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:46.011278    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z5rmx" podStartSLOduration=2.779252379 podStartE2EDuration="5.011259363s" totalImagesPullingTime="2.232006984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:46.008853417 +0000 UTC m=+83.929557469" watchObservedRunningTime="2026-07-28 10:08:46.011259363 +0000 UTC m=+83.931963415"
Jul 28 10:08:46.033710 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:46.033572    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll" podStartSLOduration=2.375803071 podStartE2EDuration="5.033555629s" totalImagesPullingTime="2.657752558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:46.032201259 +0000 UTC m=+83.952905331" watchObservedRunningTime="2026-07-28 10:08:46.033555629 +0000 UTC m=+83.954259681"
Jul 28 10:08:50.998284 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:50.998254    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8c959989b-b5krg"
Jul 28 10:08:56.041224 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.040730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-5vpqg" event={"ID":"5fef08c7-1084-4238-b972-baadace7894b","Type":"ContainerStarted","Data":"2a53f740138e6ca454862c5a0b438f7490c1184cfc47f7d5ff8b362e9bb22cff"}
Jul 28 10:08:56.041224 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.040827    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:56.042152 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.042112    2569 patch_prober.go:28] interesting pod/downloads-7c97dc4555-5vpqg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body=
Jul 28 10:08:56.042259 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.042184    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-5vpqg" podUID="5fef08c7-1084-4238-b972-baadace7894b" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused"
Jul 28 10:08:56.044123 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.044098    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"}
Jul 28 10:08:56.044257 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.044167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"}
Jul 28 10:08:56.061541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.061497    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-5vpqg" podStartSLOduration=1.5602113530000001 podStartE2EDuration="21.061467778s" totalImagesPullingTime="19.501256425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.060805382 +0000 UTC m=+93.981509436" watchObservedRunningTime="2026-07-28 10:08:56.061467778 +0000 UTC m=+93.982171811"
Jul 28 10:08:56.121758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.121730    2569 patch_prober.go:28] interesting pod/downloads-7c97dc4555-5vpqg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body=
Jul 28 10:08:56.121888 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:56.121786    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-7c97dc4555-5vpqg" podUID="5fef08c7-1084-4238-b972-baadace7894b" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused"
Jul 28 10:08:57.052877 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.052829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"}
Jul 28 10:08:57.053458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.052919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"}
Jul 28 10:08:57.053458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.052965    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"}
Jul 28 10:08:57.053458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.053008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerStarted","Data":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"}
Jul 28 10:08:57.064088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.064062    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-5vpqg"
Jul 28 10:08:57.101264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:57.101186    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.505654758 podStartE2EDuration="14.101163237s" totalImagesPullingTime="11.570693099s" totalInitContainerRuntime="24.81538ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:57.099584733 +0000 UTC m=+95.020288786" watchObservedRunningTime="2026-07-28 10:08:57.101163237 +0000 UTC m=+95.021867289"
Jul 28 10:08:58.646632 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:08:58.646566    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:01.705529 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:01.705485    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:09:01.705985 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:01.705590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:09:02.901600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:02.901563    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4rm74"
Jul 28 10:09:21.710980 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:21.710949    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:09:21.714678 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:21.714653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-847755dcc8-kl6ll"
Jul 28 10:09:43.646374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:43.646342    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:43.694084 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:43.694058    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:09:44.239075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:09:44.239050    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:01.676475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676429    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:01.677031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676852    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="prometheus" containerID="cri-o://70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e" gracePeriod=600
Jul 28 10:10:01.677031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676880    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="thanos-sidecar" containerID="cri-o://1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc" gracePeriod=600
Jul 28 10:10:01.677031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676914    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b" gracePeriod=600
Jul 28 10:10:01.677031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676913    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="config-reloader" containerID="cri-o://e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c" gracePeriod=600
Jul 28 10:10:01.677031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676889    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-web" containerID="cri-o://ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2" gracePeriod=600
Jul 28 10:10:01.677280 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.676863    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy" containerID="cri-o://c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d" gracePeriod=600
Jul 28 10:10:01.915205 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:01.915177    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.002384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002290    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002337    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002472    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002573    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002614    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002639    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002674    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002737    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002820    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002911    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002935    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q5v9\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.002992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.002963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.003381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003009    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.003381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.003381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003078    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.003381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003108    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out\") pod \"292b0de3-023f-44af-a457-680dc36bc177\" (UID: \"292b0de3-023f-44af-a457-680dc36bc177\") "
Jul 28 10:10:02.003381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003344    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.004665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003802    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.004665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.003951    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.005214 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.005185    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.005329 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.005255    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.005581 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.005557    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.006642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.006613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.006770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.006672    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.006770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.006720    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.006770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.006745    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.007280 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.007245    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:10:02.007858 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.007832    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out" (OuterVolumeSpecName: "config-out") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:10:02.007978 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.007887    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:10:02.008349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.008322    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9" (OuterVolumeSpecName: "kube-api-access-8q5v9") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "kube-api-access-8q5v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:10:02.008438 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.008404    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.008662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.008629    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:10:02.008883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.008848    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.009871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.009092    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config" (OuterVolumeSpecName: "config") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.010163 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.010142    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.020655 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.020630    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config" (OuterVolumeSpecName: "web-config") pod "292b0de3-023f-44af-a457-680dc36bc177" (UID: "292b0de3-023f-44af-a457-680dc36bc177"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:10:02.104040 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104003    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-metrics-client-ca\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104040 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104033    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-metrics-client-certs\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104040 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104044    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104053    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-web-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104063    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-tls-assets\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104070    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104078    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104086    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-grpc-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104094    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104102    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104111    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104119    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5v9\" (UniqueName: \"kubernetes.io/projected/292b0de3-023f-44af-a457-680dc36bc177-kube-api-access-8q5v9\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104127    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-db\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104134    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104142    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104150    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/292b0de3-023f-44af-a457-680dc36bc177-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104157    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/292b0de3-023f-44af-a457-680dc36bc177-config-out\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104165    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-kube-rbac-proxy\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.104264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.104174    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/292b0de3-023f-44af-a457-680dc36bc177-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294621    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294647    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294653    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294659    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294664    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294669    2569 generic.go:356] "Generic (PLEG): container finished" podID="292b0de3-023f-44af-a457-680dc36bc177" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e" exitCode=0
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"}
Jul 28 10:10:02.294735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294727    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"}
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294810    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"}
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"}
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294849    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"}
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"}
Jul 28 10:10:02.295231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.294911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"292b0de3-023f-44af-a457-680dc36bc177","Type":"ContainerDied","Data":"3f304f53fa9f3850f9edd4eb72dbad64857e20fa35fc1cf485baca353e071377"}
Jul 28 10:10:02.302787 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.302771    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.309669 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.309651    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.316315 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.316296    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.320396 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.320374    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:02.323007 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.322992    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.326720 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.326698    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:02.330311 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.330296    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.337151 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.337137    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.343529 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.343514    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.343904 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.343884    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.343975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.343913    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.343975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.343945    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.344174 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.344156    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.344217 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344183    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.344217 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344205    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.344421 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.344407    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.344467 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344425    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.344467 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344438    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.344639 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.344623    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.344678 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344642    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.344678 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344653    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.344887 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.344870    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.344927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344894    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.344927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.344909    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.345128 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.345110    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.345191 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345135    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.345191 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345159    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.345391 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:10:02.345376    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.345427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345395    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.345427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345407    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.345625 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345606    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.345665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345625    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.345847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345832    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.345891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.345848    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.346073 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346054    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.346130 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346077    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.346291 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346275    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.346339 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346291    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.346478 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346458    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.346525 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346480    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.346755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346733    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.346755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346754    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.346980 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346962    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.347074 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.346983    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.347202 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347185    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.347247 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347201    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.347388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347374    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.347433 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347388    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.347605 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347583    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.347654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347606    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.347848 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347832    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.347892 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.347848    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.348029 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348015    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.348077 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348029    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.348223 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348205    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.348288 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348224    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.348436 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348419    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.348477 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348436    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.348635 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348621    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.348672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348635    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.348861 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348843    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.348937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.348864    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.349089 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349069    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.349128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349096    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.349291 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349277    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.349337 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349290    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.349505 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349487    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.349552 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349505    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349661    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.349755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349676    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.349893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349863    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.349893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.349891    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.350124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350107    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.350173 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350125    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.350333 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350310    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.350376 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350333    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.350571 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350538    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.350619 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350571    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.354796 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.350792    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.354796 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.351123    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.355171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355150    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.355218 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355174    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.355412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355397    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.355459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355413    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.355646 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355628    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.355710 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355648    2569 scope.go:117] "RemoveContainer" containerID="fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"
Jul 28 10:10:02.355902 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355870    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b"} err="failed to get container status \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": rpc error: code = NotFound desc = could not find container \"fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b\": container with ID starting with fd0a532522fb9b5c9769bfe21e17278a52e6d1bdeb49aec489fe5df0599d861b not found: ID does not exist"
Jul 28 10:10:02.355902 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.355896    2569 scope.go:117] "RemoveContainer" containerID="c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"
Jul 28 10:10:02.356101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356083    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d"} err="failed to get container status \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": rpc error: code = NotFound desc = could not find container \"c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d\": container with ID starting with c156e892e3bdc5feea24f5ceb0399e3aef458ae45bf114b61f9b8a667a8c983d not found: ID does not exist"
Jul 28 10:10:02.356167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356103    2569 scope.go:117] "RemoveContainer" containerID="ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"
Jul 28 10:10:02.356356 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356339    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2"} err="failed to get container status \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": rpc error: code = NotFound desc = could not find container \"ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2\": container with ID starting with ad5f516d02d3f62278ccf7424bf0287cb67071bde511d9edcb4a2c4fff94d6a2 not found: ID does not exist"
Jul 28 10:10:02.356409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356356    2569 scope.go:117] "RemoveContainer" containerID="1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"
Jul 28 10:10:02.356626 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356603    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc"} err="failed to get container status \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": rpc error: code = NotFound desc = could not find container \"1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc\": container with ID starting with 1c1d97988866884bd731e222be02d5d83e94e263e2f5b010b6c3de781a7ae1fc not found: ID does not exist"
Jul 28 10:10:02.356674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356633    2569 scope.go:117] "RemoveContainer" containerID="e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"
Jul 28 10:10:02.356989 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356953    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c"} err="failed to get container status \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": rpc error: code = NotFound desc = could not find container \"e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c\": container with ID starting with e9a9f85a58ac86e03e13d12185b758f09a7160e9a76e87f52380212176e9427c not found: ID does not exist"
Jul 28 10:10:02.356989 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.356989    2569 scope.go:117] "RemoveContainer" containerID="70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"
Jul 28 10:10:02.357209 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.357191    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e"} err="failed to get container status \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": rpc error: code = NotFound desc = could not find container \"70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e\": container with ID starting with 70468ab2d639041962742a38b6837b25ce25be7688ebe085d2395b61f0debc5e not found: ID does not exist"
Jul 28 10:10:02.357275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.357211    2569 scope.go:117] "RemoveContainer" containerID="862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"
Jul 28 10:10:02.357420 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.357402    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e"} err="failed to get container status \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": rpc error: code = NotFound desc = could not find container \"862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e\": container with ID starting with 862a76906121f174cb5daaa1a55ddea35c9cc2ff1fe9b64c5054fa449dad2e9e not found: ID does not exist"
Jul 28 10:10:02.398649 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398614    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:02.398928 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398915    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="init-config-reloader"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398929    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="init-config-reloader"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398945    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="prometheus"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398951    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="prometheus"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398957    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-thanos"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398962    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-thanos"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398970    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="thanos-sidecar"
Jul 28 10:10:02.398974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398975    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="thanos-sidecar"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398982    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="config-reloader"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398988    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="config-reloader"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398994    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.398999    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399005    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-web"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399010    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-web"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399081    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-thanos"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399088    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy-web"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399093    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="config-reloader"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399098    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="kube-rbac-proxy"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399104    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="prometheus"
Jul 28 10:10:02.399177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.399109    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="292b0de3-023f-44af-a457-680dc36bc177" containerName="thanos-sidecar"
Jul 28 10:10:02.402876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.402855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.405527 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.405506    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 10:10:02.405823 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.405805    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 10:10:02.405923 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.405858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 10:10:02.406136 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.406116    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xzkds\""
Jul 28 10:10:02.406713 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.406696    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 10:10:02.406806 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.406775    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 10:10:02.406905 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.406892    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 10:10:02.407437 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407411    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 10:10:02.407541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407450    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ds848bs3kce69\""
Jul 28 10:10:02.407541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407457    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 10:10:02.407701 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407624    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 10:10:02.408049 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407851    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 10:10:02.408049 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.407922    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 10:10:02.410524 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.408569    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 10:10:02.411904 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.411884    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 10:10:02.412863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.412845    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 10:10:02.420552 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.420527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:02.507539 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507539 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rvb\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-kube-api-access-s9rvb\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507603    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507618    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-config-out\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507726    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.507822 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-web-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507885    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507918    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.507998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.508016    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.508106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.508030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.567229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.567151    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292b0de3-023f-44af-a457-680dc36bc177" path="/var/lib/kubelet/pods/292b0de3-023f-44af-a457-680dc36bc177/volumes"
Jul 28 10:10:02.608929 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.608883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-web-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.608951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.608977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.608997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609325 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609375 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609324    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609375 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609392    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609424    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rvb\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-kube-api-access-s9rvb\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-config-out\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609726    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.609937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.609755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.611402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.610724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.611402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.610922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.611402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.611167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.611402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.611361    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.611967 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.611940    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.612152 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.612128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.613223 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.613175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.613356 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.613341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-web-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.613537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.613510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a917ac49-b6a3-44f4-80fd-4276875ac989-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.613858 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.613818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.613943 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.613879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.614083 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.614061    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.614235 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.614221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.614400 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.614383    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-config\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.614856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.614837    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a917ac49-b6a3-44f4-80fd-4276875ac989-config-out\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.614902 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.614872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a917ac49-b6a3-44f4-80fd-4276875ac989-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.615051 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.615038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.618414 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.618397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rvb\" (UniqueName: \"kubernetes.io/projected/a917ac49-b6a3-44f4-80fd-4276875ac989-kube-api-access-s9rvb\") pod \"prometheus-k8s-0\" (UID: \"a917ac49-b6a3-44f4-80fd-4276875ac989\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.714700 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.714650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:10:02.850442 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:02.850297    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:10:02.853240 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:10:02.853208    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda917ac49_b6a3_44f4_80fd_4276875ac989.slice/crio-8e655856a8dc4b8f6fd8032509aca231b272bb8f26626551914385e4dab022dd WatchSource:0}: Error finding container 8e655856a8dc4b8f6fd8032509aca231b272bb8f26626551914385e4dab022dd: Status 404 returned error can't find the container with id 8e655856a8dc4b8f6fd8032509aca231b272bb8f26626551914385e4dab022dd
Jul 28 10:10:03.301515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:03.301479    2569 generic.go:356] "Generic (PLEG): container finished" podID="a917ac49-b6a3-44f4-80fd-4276875ac989" containerID="b26d4cd199c3ab0eb18d2c743891daff1586a98215f4ad38a25d78425656bed8" exitCode=0
Jul 28 10:10:03.301515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:03.301516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerDied","Data":"b26d4cd199c3ab0eb18d2c743891daff1586a98215f4ad38a25d78425656bed8"}
Jul 28 10:10:03.301800 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:03.301564    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"8e655856a8dc4b8f6fd8032509aca231b272bb8f26626551914385e4dab022dd"}
Jul 28 10:10:04.309031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.308997    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"8bc88f6856e8af16b4a96745c13e71e9c91f8fc9abd020354f1585a0b3cd4a39"}
Jul 28 10:10:04.309387 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.309062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"caea45fa2620e8b47623b04ded5c75968a178f0ada2a534f6a0447555d61be0d"}
Jul 28 10:10:04.309387 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.309093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"7de399213ad3835f06fdcc4449db813e6f31290d7c645c004cb0724bf5553a50"}
Jul 28 10:10:04.309387 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.309117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"783d5107da3443bb4d3d65be11d91cd9e966fb2bee46a7ce5cc38139fcfd9dd0"}
Jul 28 10:10:04.309387 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.309138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"b034ac9b9bae8b96d99940599ae5f1132b849482a08250da6b52ee05482f6b80"}
Jul 28 10:10:04.309387 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.309159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a917ac49-b6a3-44f4-80fd-4276875ac989","Type":"ContainerStarted","Data":"c03b21559cc5b87273bfc02e6617a7ac25e461d639346dd47812cf9ff376d157"}
Jul 28 10:10:04.338736 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:04.338662    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.321744809 podStartE2EDuration="2.338644649s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.89984ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:04.336103348 +0000 UTC m=+162.256807398" watchObservedRunningTime="2026-07-28 10:10:04.338644649 +0000 UTC m=+162.259348702"
Jul 28 10:10:07.715797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:10:07.715736    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:11:02.715410 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:11:02.715371    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:11:02.730325 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:11:02.730301    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:11:03.543421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:11:03.543392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:12:22.395732 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:12:22.395675    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:12:22.399851 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:12:22.399824    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:12:22.402700 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:12:22.402667    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:14:59.494809 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.494770    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-gwkp4"]
Jul 28 10:14:59.496723 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.496704    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.499159 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.499136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 10:14:59.499159 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.499136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 10:14:59.499478 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.499459    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 10:14:59.500282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.500263    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bq7d8\""
Jul 28 10:14:59.509464 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.509436    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gwkp4"]
Jul 28 10:14:59.511308 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.511283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58rpg\" (UniqueName: \"kubernetes.io/projected/358b6648-fa79-49f8-bcb1-f5708ba5aa10-kube-api-access-58rpg\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.511414 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.511373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/358b6648-fa79-49f8-bcb1-f5708ba5aa10-data\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.612115 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.612078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/358b6648-fa79-49f8-bcb1-f5708ba5aa10-data\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.612297 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.612132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58rpg\" (UniqueName: \"kubernetes.io/projected/358b6648-fa79-49f8-bcb1-f5708ba5aa10-kube-api-access-58rpg\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.612446 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.612426    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/358b6648-fa79-49f8-bcb1-f5708ba5aa10-data\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.621168 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.621143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58rpg\" (UniqueName: \"kubernetes.io/projected/358b6648-fa79-49f8-bcb1-f5708ba5aa10-kube-api-access-58rpg\") pod \"seaweedfs-64568bcd49-gwkp4\" (UID: \"358b6648-fa79-49f8-bcb1-f5708ba5aa10\") " pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.805867 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.805774    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:14:59.941082 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.941046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-gwkp4"]
Jul 28 10:14:59.944247 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:14:59.944221    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358b6648_fa79_49f8_bcb1_f5708ba5aa10.slice/crio-684d0ee82ea4558b240c2fcdf570df13650bef4cc32f6c2517cbcb30e272dfe0 WatchSource:0}: Error finding container 684d0ee82ea4558b240c2fcdf570df13650bef4cc32f6c2517cbcb30e272dfe0: Status 404 returned error can't find the container with id 684d0ee82ea4558b240c2fcdf570df13650bef4cc32f6c2517cbcb30e272dfe0
Jul 28 10:14:59.945544 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:14:59.945530    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:15:00.361373 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:15:00.361342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gwkp4" event={"ID":"358b6648-fa79-49f8-bcb1-f5708ba5aa10","Type":"ContainerStarted","Data":"684d0ee82ea4558b240c2fcdf570df13650bef4cc32f6c2517cbcb30e272dfe0"}
Jul 28 10:15:04.377460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:15:04.377424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-gwkp4" event={"ID":"358b6648-fa79-49f8-bcb1-f5708ba5aa10","Type":"ContainerStarted","Data":"2716166f8578d1c0b6a2838959a510f5c8231787a36b880575f38cd8cce4f5ab"}
Jul 28 10:15:04.377853 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:15:04.377530    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:15:04.397456 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:15:04.397408    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-gwkp4" podStartSLOduration=2.016036972 podStartE2EDuration="5.397392656s" totalImagesPullingTime="3.381355684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:04.395712173 +0000 UTC m=+462.316416221" watchObservedRunningTime="2026-07-28 10:15:04.397392656 +0000 UTC m=+462.318096708"
Jul 28 10:15:10.382848 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:15:10.382819    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-gwkp4"
Jul 28 10:16:33.142499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.142413    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2hw6q"]
Jul 28 10:16:33.144550 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.144534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.154248 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.154221    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2hw6q"]
Jul 28 10:16:33.265097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.265061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.265241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.265112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxf6\" (UniqueName: \"kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.365588 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.365556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.365741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.365605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxf6\" (UniqueName: \"kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.366009 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.365989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.375012 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.374990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxf6\" (UniqueName: \"kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6\") pod \"s3-init-2hw6q\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") " pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.462862 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.462835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2hw6q"
Jul 28 10:16:33.583725 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.583672    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2hw6q"]
Jul 28 10:16:33.585994 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:16:33.585965    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d8076f_30dd_4054_8c4f_5d35e38cc64c.slice/crio-84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680 WatchSource:0}: Error finding container 84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680: Status 404 returned error can't find the container with id 84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680
Jul 28 10:16:33.699361 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:33.699318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerStarted","Data":"84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680"}
Jul 28 10:16:37.722123 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:37.722037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerStarted","Data":"582d94da6dc57ac28f0c3a3a3a7edfa4743d630fa404176a0ce409eb9ad21275"}
Jul 28 10:16:39.731051 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:39.731019    2569 generic.go:356] "Generic (PLEG): container finished" podID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerID="582d94da6dc57ac28f0c3a3a3a7edfa4743d630fa404176a0ce409eb9ad21275" exitCode=0
Jul 28 10:16:39.731051 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:39.731053    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerDied","Data":"582d94da6dc57ac28f0c3a3a3a7edfa4743d630fa404176a0ce409eb9ad21275"}
Jul 28 10:16:44.754134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:44.754093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerStarted","Data":"a8577fea817e8f7f0a241e68983302ce0dfc4e78c956d27e237aef5c3e958a1c"}
Jul 28 10:16:44.770903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:16:44.770829    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-2hw6q" podStartSLOduration=2.111982751 podStartE2EDuration="11.770809809s" totalImagesPullingTime="8.150620071s" totalInitContainerRuntime="1.508206987s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:44.770712968 +0000 UTC m=+562.691417021" watchObservedRunningTime="2026-07-28 10:16:44.770809809 +0000 UTC m=+562.691513860"
Jul 28 10:17:22.420596 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:22.420566    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:17:22.422272 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:22.422253    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:17:48.996680 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:48.996646    2569 generic.go:356] "Generic (PLEG): container finished" podID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerID="a8577fea817e8f7f0a241e68983302ce0dfc4e78c956d27e237aef5c3e958a1c" exitCode=0
Jul 28 10:17:48.997108 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:48.996714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerDied","Data":"a8577fea817e8f7f0a241e68983302ce0dfc4e78c956d27e237aef5c3e958a1c"}
Jul 28 10:17:50.119381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.119356    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2hw6q"
Jul 28 10:17:50.291863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.291769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlxf6\" (UniqueName: \"kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6\") pod \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") "
Jul 28 10:17:50.291863 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.291819    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache\") pod \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\" (UID: \"e4d8076f-30dd-4054-8c4f-5d35e38cc64c\") "
Jul 28 10:17:50.293981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.293956    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6" (OuterVolumeSpecName: "kube-api-access-dlxf6") pod "e4d8076f-30dd-4054-8c4f-5d35e38cc64c" (UID: "e4d8076f-30dd-4054-8c4f-5d35e38cc64c"). InnerVolumeSpecName "kube-api-access-dlxf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:50.314226 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.314192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4d8076f-30dd-4054-8c4f-5d35e38cc64c" (UID: "e4d8076f-30dd-4054-8c4f-5d35e38cc64c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:50.392550 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.392521    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxf6\" (UniqueName: \"kubernetes.io/projected/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-kube-api-access-dlxf6\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:17:50.392550 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:50.392544    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d8076f-30dd-4054-8c4f-5d35e38cc64c-model-cache\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:17:51.006092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.006062    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2hw6q"
Jul 28 10:17:51.006092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.006076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2hw6q" event={"ID":"e4d8076f-30dd-4054-8c4f-5d35e38cc64c","Type":"ContainerDied","Data":"84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680"}
Jul 28 10:17:51.006297 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.006110    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84c74c328ab8574832df3e39f72c33c79e72c93fa385073077799c949eb4e680"
Jul 28 10:17:51.790301 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.790246    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:51.790926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.790901    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerName="download-hf-model"
Jul 28 10:17:51.791000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.790929    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerName="download-hf-model"
Jul 28 10:17:51.791000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.790943    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerName="s3-init"
Jul 28 10:17:51.791000 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.790951    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerName="s3-init"
Jul 28 10:17:51.791140 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.791104    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d8076f-30dd-4054-8c4f-5d35e38cc64c" containerName="s3-init"
Jul 28 10:17:51.793636 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.793608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:51.796080 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.796061    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 10:17:51.798796 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.798770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:51.905392 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.905349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:51.905573 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:51.905401    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56xv\" (UniqueName: \"kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.006291 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.006256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56xv\" (UniqueName: \"kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.006446 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.006353    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.006750 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.006730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.015358 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.015335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56xv\" (UniqueName: \"kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv\") pod \"seaweedfs-tls-custom-6644bf5d6c-qrfqk\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.103300 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.103217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:52.225212 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:52.225187    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:52.227698 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:17:52.227654    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1860606_8b8a_4db3_8374_cac8c059c727.slice/crio-6983a2eedee91ba765b3dfbf3df6a856077beddcfb0f39186a313d020b218d3c WatchSource:0}: Error finding container 6983a2eedee91ba765b3dfbf3df6a856077beddcfb0f39186a313d020b218d3c: Status 404 returned error can't find the container with id 6983a2eedee91ba765b3dfbf3df6a856077beddcfb0f39186a313d020b218d3c
Jul 28 10:17:53.014698 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:53.014649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" event={"ID":"e1860606-8b8a-4db3-8374-cac8c059c727","Type":"ContainerStarted","Data":"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"}
Jul 28 10:17:53.015051 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:53.014724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" event={"ID":"e1860606-8b8a-4db3-8374-cac8c059c727","Type":"ContainerStarted","Data":"6983a2eedee91ba765b3dfbf3df6a856077beddcfb0f39186a313d020b218d3c"}
Jul 28 10:17:53.031903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:53.031852    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" podStartSLOduration=1.737639148 podStartE2EDuration="2.031834943s" totalImagesPullingTime="294.195795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:53.031141123 +0000 UTC m=+630.951845175" watchObservedRunningTime="2026-07-28 10:17:53.031834943 +0000 UTC m=+630.952538995"
Jul 28 10:17:54.168727 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:54.168678    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:55.021162 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:55.021120    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" podUID="e1860606-8b8a-4db3-8374-cac8c059c727" containerName="seaweedfs-tls-custom" containerID="cri-o://a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a" gracePeriod=30
Jul 28 10:17:56.259989 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.259965    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:56.336785 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.336701    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data\") pod \"e1860606-8b8a-4db3-8374-cac8c059c727\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") "
Jul 28 10:17:56.337920 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.337897    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data" (OuterVolumeSpecName: "data") pod "e1860606-8b8a-4db3-8374-cac8c059c727" (UID: "e1860606-8b8a-4db3-8374-cac8c059c727"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:17:56.437885 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.437844    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s56xv\" (UniqueName: \"kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv\") pod \"e1860606-8b8a-4db3-8374-cac8c059c727\" (UID: \"e1860606-8b8a-4db3-8374-cac8c059c727\") "
Jul 28 10:17:56.438008 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.437999    2569 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1860606-8b8a-4db3-8374-cac8c059c727-data\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:17:56.439814 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.439783    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv" (OuterVolumeSpecName: "kube-api-access-s56xv") pod "e1860606-8b8a-4db3-8374-cac8c059c727" (UID: "e1860606-8b8a-4db3-8374-cac8c059c727"). InnerVolumeSpecName "kube-api-access-s56xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:56.538874 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:56.538836    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56xv\" (UniqueName: \"kubernetes.io/projected/e1860606-8b8a-4db3-8374-cac8c059c727-kube-api-access-s56xv\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:17:57.033883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.033850    2569 generic.go:356] "Generic (PLEG): container finished" podID="e1860606-8b8a-4db3-8374-cac8c059c727" containerID="a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a" exitCode=0
Jul 28 10:17:57.034043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.033927    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"
Jul 28 10:17:57.034043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.033934    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" event={"ID":"e1860606-8b8a-4db3-8374-cac8c059c727","Type":"ContainerDied","Data":"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"}
Jul 28 10:17:57.034043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.034008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk" event={"ID":"e1860606-8b8a-4db3-8374-cac8c059c727","Type":"ContainerDied","Data":"6983a2eedee91ba765b3dfbf3df6a856077beddcfb0f39186a313d020b218d3c"}
Jul 28 10:17:57.034043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.034028    2569 scope.go:117] "RemoveContainer" containerID="a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"
Jul 28 10:17:57.043216 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.043198    2569 scope.go:117] "RemoveContainer" containerID="a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"
Jul 28 10:17:57.043491 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:17:57.043468    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a\": container with ID starting with a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a not found: ID does not exist" containerID="a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"
Jul 28 10:17:57.043579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.043501    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a"} err="failed to get container status \"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a\": rpc error: code = NotFound desc = could not find container \"a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a\": container with ID starting with a0300030b6e8997d160604023eee0683d4683fd488684a657e55a84466afca2a not found: ID does not exist"
Jul 28 10:17:57.053353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.053326    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:57.056214 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.056191    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-qrfqk"]
Jul 28 10:17:57.081932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.081906    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"]
Jul 28 10:17:57.082182 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.082171    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1860606-8b8a-4db3-8374-cac8c059c727" containerName="seaweedfs-tls-custom"
Jul 28 10:17:57.082225 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.082184    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1860606-8b8a-4db3-8374-cac8c059c727" containerName="seaweedfs-tls-custom"
Jul 28 10:17:57.082265 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.082236    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1860606-8b8a-4db3-8374-cac8c059c727" containerName="seaweedfs-tls-custom"
Jul 28 10:17:57.086389 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.086373    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.088937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.088916    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 10:17:57.089047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.089002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 28 10:17:57.094397 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.094375    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"]
Jul 28 10:17:57.142766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.142734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2b8\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-kube-api-access-qz2b8\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.142927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.142780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.142927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.142802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de01ce05-d9ec-48ad-9ad7-de1038191542-data\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.243556 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.243518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2b8\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-kube-api-access-qz2b8\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.243556 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.243558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.243780 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.243579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de01ce05-d9ec-48ad-9ad7-de1038191542-data\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.243973 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.243955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/de01ce05-d9ec-48ad-9ad7-de1038191542-data\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.246017 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.245989    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.251314 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.251293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2b8\" (UniqueName: \"kubernetes.io/projected/de01ce05-d9ec-48ad-9ad7-de1038191542-kube-api-access-qz2b8\") pod \"seaweedfs-tls-custom-8b44d767d-wt9bz\" (UID: \"de01ce05-d9ec-48ad-9ad7-de1038191542\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.395733 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.395704    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"
Jul 28 10:17:57.519569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:57.519536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wt9bz"]
Jul 28 10:17:57.522762 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:17:57.522730    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde01ce05_d9ec_48ad_9ad7_de1038191542.slice/crio-817af4d315288f00a22087da162b07397b7d29b142028b166cf32126e0774b79 WatchSource:0}: Error finding container 817af4d315288f00a22087da162b07397b7d29b142028b166cf32126e0774b79: Status 404 returned error can't find the container with id 817af4d315288f00a22087da162b07397b7d29b142028b166cf32126e0774b79
Jul 28 10:17:58.038916 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.038815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz" event={"ID":"de01ce05-d9ec-48ad-9ad7-de1038191542","Type":"ContainerStarted","Data":"f5280fd0ef49ecb85679168dc7acc24718f30821cc23b8c28b9f2b6550d65ceb"}
Jul 28 10:17:58.038916 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.038883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz" event={"ID":"de01ce05-d9ec-48ad-9ad7-de1038191542","Type":"ContainerStarted","Data":"817af4d315288f00a22087da162b07397b7d29b142028b166cf32126e0774b79"}
Jul 28 10:17:58.060297 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.060247    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-wt9bz" podStartSLOduration=0.820808884 podStartE2EDuration="1.060233055s" totalImagesPullingTime="239.424171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:58.058677353 +0000 UTC m=+635.979381406" watchObservedRunningTime="2026-07-28 10:17:58.060233055 +0000 UTC m=+635.980937107"
Jul 28 10:17:58.373903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.373819    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-w828m"]
Jul 28 10:17:58.377043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.377026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.385463 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.385441    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-w828m"]
Jul 28 10:17:58.452731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.452697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfrb\" (UniqueName: \"kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.453090 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.452745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.553536 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.553502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.553676 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.553570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfrb\" (UniqueName: \"kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.553880 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.553862    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.562908 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.562882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfrb\" (UniqueName: \"kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb\") pod \"s3-tls-init-custom-w828m\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") " pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.566038 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.566015    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1860606-8b8a-4db3-8374-cac8c059c727" path="/var/lib/kubelet/pods/e1860606-8b8a-4db3-8374-cac8c059c727/volumes"
Jul 28 10:17:58.704062 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.704026    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:17:58.832810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:58.832787    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-w828m"]
Jul 28 10:17:58.835270 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:17:58.835237    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode14a887b_e068_4384_91f6_42bd8c16ca18.slice/crio-9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d WatchSource:0}: Error finding container 9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d: Status 404 returned error can't find the container with id 9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d
Jul 28 10:17:59.045652 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:59.045562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerStarted","Data":"01b1a964331744ae2162e3b40a1a2636ec8774fd97392faacbbd341f002545ae"}
Jul 28 10:17:59.045652 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:17:59.045610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerStarted","Data":"9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d"}
Jul 28 10:18:01.057573 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:01.057537    2569 generic.go:356] "Generic (PLEG): container finished" podID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerID="01b1a964331744ae2162e3b40a1a2636ec8774fd97392faacbbd341f002545ae" exitCode=0
Jul 28 10:18:01.059906 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:01.057613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerDied","Data":"01b1a964331744ae2162e3b40a1a2636ec8774fd97392faacbbd341f002545ae"}
Jul 28 10:18:02.063065 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:02.063032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerStarted","Data":"bf80378593f5d23d878aaa10a21efa4a62a738ce760b3096799164bef08b5145"}
Jul 28 10:18:02.081014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:02.080937    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-w828m" podStartSLOduration=2.555883738 podStartE2EDuration="4.080919771s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.525036033s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:02.080739392 +0000 UTC m=+640.001443444" watchObservedRunningTime="2026-07-28 10:18:02.080919771 +0000 UTC m=+640.001623823"
Jul 28 10:18:08.091322 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:08.091280    2569 generic.go:356] "Generic (PLEG): container finished" podID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerID="bf80378593f5d23d878aaa10a21efa4a62a738ce760b3096799164bef08b5145" exitCode=0
Jul 28 10:18:08.091750 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:08.091323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerDied","Data":"bf80378593f5d23d878aaa10a21efa4a62a738ce760b3096799164bef08b5145"}
Jul 28 10:18:09.236603 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.236577    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:18:09.339015 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.338990    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache\") pod \"e14a887b-e068-4384-91f6-42bd8c16ca18\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") "
Jul 28 10:18:09.339122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.339031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqfrb\" (UniqueName: \"kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb\") pod \"e14a887b-e068-4384-91f6-42bd8c16ca18\" (UID: \"e14a887b-e068-4384-91f6-42bd8c16ca18\") "
Jul 28 10:18:09.341073 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.341044    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb" (OuterVolumeSpecName: "kube-api-access-fqfrb") pod "e14a887b-e068-4384-91f6-42bd8c16ca18" (UID: "e14a887b-e068-4384-91f6-42bd8c16ca18"). InnerVolumeSpecName "kube-api-access-fqfrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:18:09.360427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.360352    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache" (OuterVolumeSpecName: "model-cache") pod "e14a887b-e068-4384-91f6-42bd8c16ca18" (UID: "e14a887b-e068-4384-91f6-42bd8c16ca18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:18:09.440065 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.440032    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14a887b-e068-4384-91f6-42bd8c16ca18-model-cache\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:18:09.440065 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:09.440057    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfrb\" (UniqueName: \"kubernetes.io/projected/e14a887b-e068-4384-91f6-42bd8c16ca18-kube-api-access-fqfrb\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:18:10.100395 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.100370    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w828m"
Jul 28 10:18:10.100562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.100366    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w828m" event={"ID":"e14a887b-e068-4384-91f6-42bd8c16ca18","Type":"ContainerDied","Data":"9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d"}
Jul 28 10:18:10.100562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.100486    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9daae1ea53c05bd2c96fb2299e2a18900d3a0f2bf7ccf0a31c20c817b2d2d60d"
Jul 28 10:18:10.741477 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741445    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"]
Jul 28 10:18:10.741926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741765    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerName="s3-tls-init-custom"
Jul 28 10:18:10.741926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741777    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerName="s3-tls-init-custom"
Jul 28 10:18:10.741926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741790    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerName="download-hf-model"
Jul 28 10:18:10.741926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741797    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerName="download-hf-model"
Jul 28 10:18:10.741926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.741861    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14a887b-e068-4384-91f6-42bd8c16ca18" containerName="s3-tls-init-custom"
Jul 28 10:18:10.743804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.743786    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.747209 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.747185    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 10:18:10.747732 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.747714    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 10:18:10.757354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.757332    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"]
Jul 28 10:18:10.853410 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.853367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.853410 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.853412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7lt\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-kube-api-access-xc7lt\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.853645 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.853436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04cafd22-b69e-4346-8d76-8e4339203d04-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.954838 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.954793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04cafd22-b69e-4346-8d76-8e4339203d04-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.955014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.954912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.955014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.954932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7lt\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-kube-api-access-xc7lt\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.955090 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:18:10.955039    2569 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 28 10:18:10.955090 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:18:10.955058    2569 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-4wplm: secret "seaweedfs-tls-serving" not found
Jul 28 10:18:10.955157 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:18:10.955130    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving podName:04cafd22-b69e-4346-8d76-8e4339203d04 nodeName:}" failed. No retries permitted until 2026-07-28 10:18:11.455109423 +0000 UTC m=+649.375813467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-4wplm" (UID: "04cafd22-b69e-4346-8d76-8e4339203d04") : secret "seaweedfs-tls-serving" not found
Jul 28 10:18:10.955197 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.955170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/04cafd22-b69e-4346-8d76-8e4339203d04-data\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:10.963474 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:10.963446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7lt\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-kube-api-access-xc7lt\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:11.459120 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:11.459069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:11.461425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:11.461400    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/04cafd22-b69e-4346-8d76-8e4339203d04-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-4wplm\" (UID: \"04cafd22-b69e-4346-8d76-8e4339203d04\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:11.652896 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:11.652842    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"
Jul 28 10:18:11.781912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:11.781882    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-4wplm"]
Jul 28 10:18:11.784772 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:18:11.784746    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cafd22_b69e_4346_8d76_8e4339203d04.slice/crio-3ce27d6e1be7ec233615a01a6022759ea9436a0ddebacae560102b7eeabfe4c2 WatchSource:0}: Error finding container 3ce27d6e1be7ec233615a01a6022759ea9436a0ddebacae560102b7eeabfe4c2: Status 404 returned error can't find the container with id 3ce27d6e1be7ec233615a01a6022759ea9436a0ddebacae560102b7eeabfe4c2
Jul 28 10:18:12.110067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.110022    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm" event={"ID":"04cafd22-b69e-4346-8d76-8e4339203d04","Type":"ContainerStarted","Data":"27c3b787ce9a9c032d7fdf501d88a0e2b9509652393d8b6de52bc050a153d4eb"}
Jul 28 10:18:12.110242 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.110086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm" event={"ID":"04cafd22-b69e-4346-8d76-8e4339203d04","Type":"ContainerStarted","Data":"3ce27d6e1be7ec233615a01a6022759ea9436a0ddebacae560102b7eeabfe4c2"}
Jul 28 10:18:12.130345 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.130290    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-4wplm" podStartSLOduration=1.903874843 podStartE2EDuration="2.130274811s" totalImagesPullingTime="226.399968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:12.129564482 +0000 UTC m=+650.050268534" watchObservedRunningTime="2026-07-28 10:18:12.130274811 +0000 UTC m=+650.050978876"
Jul 28 10:18:12.693467 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.693433    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-fqkpm"]
Jul 28 10:18:12.695485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.695469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.707040 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.707015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fqkpm"]
Jul 28 10:18:12.771057 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.771021    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.771057 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.771072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd48v\" (UniqueName: \"kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.871497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.871464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.871987 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.871507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd48v\" (UniqueName: \"kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.872046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.871986    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:12.881082 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:12.881059    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd48v\" (UniqueName: \"kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v\") pod \"s3-tls-init-serving-fqkpm\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") " pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:13.015246 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:13.015165    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:13.140604 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:13.140578    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-fqkpm"]
Jul 28 10:18:13.142675 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:18:13.142641    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54cdfa2e_c983_4ac9_a04e_f63bda76d19e.slice/crio-da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b WatchSource:0}: Error finding container da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b: Status 404 returned error can't find the container with id da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b
Jul 28 10:18:14.122025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:14.121992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerStarted","Data":"c72a355246d2a8d0c5523e1b732e6dd424dbc6ea40aa601000b1aee7fdd45ce7"}
Jul 28 10:18:14.122393 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:14.122036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerStarted","Data":"da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b"}
Jul 28 10:18:15.126820 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:15.126742    2569 generic.go:356] "Generic (PLEG): container finished" podID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerID="c72a355246d2a8d0c5523e1b732e6dd424dbc6ea40aa601000b1aee7fdd45ce7" exitCode=0
Jul 28 10:18:15.127189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:15.126827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerDied","Data":"c72a355246d2a8d0c5523e1b732e6dd424dbc6ea40aa601000b1aee7fdd45ce7"}
Jul 28 10:18:16.135644 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:16.135601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerStarted","Data":"9fc70cccd3ae27d654ba164922b37cfd0860d58d6852b0eac133de07696fce0a"}
Jul 28 10:18:16.155769 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:16.155712    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-fqkpm" podStartSLOduration=2.559483024 podStartE2EDuration="4.155679313s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.596196289s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:16.155294255 +0000 UTC m=+654.075998307" watchObservedRunningTime="2026-07-28 10:18:16.155679313 +0000 UTC m=+654.076383366"
Jul 28 10:18:22.162063 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:22.162032    2569 generic.go:356] "Generic (PLEG): container finished" podID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerID="9fc70cccd3ae27d654ba164922b37cfd0860d58d6852b0eac133de07696fce0a" exitCode=0
Jul 28 10:18:22.162544 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:22.162105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerDied","Data":"9fc70cccd3ae27d654ba164922b37cfd0860d58d6852b0eac133de07696fce0a"}
Jul 28 10:18:23.312924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.312897    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:23.363020 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.362986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache\") pod \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") "
Jul 28 10:18:23.363168 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.363027    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd48v\" (UniqueName: \"kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v\") pod \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\" (UID: \"54cdfa2e-c983-4ac9-a04e-f63bda76d19e\") "
Jul 28 10:18:23.365118 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.365089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v" (OuterVolumeSpecName: "kube-api-access-dd48v") pod "54cdfa2e-c983-4ac9-a04e-f63bda76d19e" (UID: "54cdfa2e-c983-4ac9-a04e-f63bda76d19e"). InnerVolumeSpecName "kube-api-access-dd48v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:18:23.385864 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.385828    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache" (OuterVolumeSpecName: "model-cache") pod "54cdfa2e-c983-4ac9-a04e-f63bda76d19e" (UID: "54cdfa2e-c983-4ac9-a04e-f63bda76d19e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:18:23.464616 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.464548    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-model-cache\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:18:23.464616 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:23.464570    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd48v\" (UniqueName: \"kubernetes.io/projected/54cdfa2e-c983-4ac9-a04e-f63bda76d19e-kube-api-access-dd48v\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:18:24.170670 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:24.170637    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-fqkpm"
Jul 28 10:18:24.170670 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:24.170655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-fqkpm" event={"ID":"54cdfa2e-c983-4ac9-a04e-f63bda76d19e","Type":"ContainerDied","Data":"da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b"}
Jul 28 10:18:24.170889 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:24.170704    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da45097c84309ebea12fa45ca2d97222090fabd18ea7b980237bbcdb7e47bb5b"
Jul 28 10:18:36.384707 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.384650    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:18:36.385081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.384961    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerName="s3-tls-init-serving"
Jul 28 10:18:36.385081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.384974    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerName="s3-tls-init-serving"
Jul 28 10:18:36.385081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.384993    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerName="download-hf-model"
Jul 28 10:18:36.385081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.384999    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerName="download-hf-model"
Jul 28 10:18:36.385081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.385047    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cdfa2e-c983-4ac9-a04e-f63bda76d19e" containerName="s3-tls-init-serving"
Jul 28 10:18:36.390080 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.390061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.396103 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396076    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:18:36.396264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396246    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:18:36.396330 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396302    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:18:36.396847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396824    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:18:36.396941 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396853    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:18:36.396941 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 10:18:36.396941 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.396869    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 10:18:36.402134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.402112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:18:36.468840 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.468798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.469014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.468844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnmg\" (UniqueName: \"kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.469014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.468916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.469106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.469018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.469106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.469060    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.569434 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.569617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.569617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnmg\" (UniqueName: \"kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.569617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.569617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.570001 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.569982    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.570278 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.570257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.570323 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.570257    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.571895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.571873    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.586935 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.586905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnmg\" (UniqueName: \"kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg\") pod \"isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.700229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.700194    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:36.829196 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:36.829171    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:18:36.831749 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:18:36.831723    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6125c782_f1be_4026_a953_12fa5000e291.slice/crio-f6119d0296daf6b349c126be282a4a3d0c8b78f4d0c3eb84b61bc5b805232aef WatchSource:0}: Error finding container f6119d0296daf6b349c126be282a4a3d0c8b78f4d0c3eb84b61bc5b805232aef: Status 404 returned error can't find the container with id f6119d0296daf6b349c126be282a4a3d0c8b78f4d0c3eb84b61bc5b805232aef
Jul 28 10:18:37.223047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:37.223007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerStarted","Data":"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"}
Jul 28 10:18:37.223329 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:37.223121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerStarted","Data":"f6119d0296daf6b349c126be282a4a3d0c8b78f4d0c3eb84b61bc5b805232aef"}
Jul 28 10:18:41.240394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:41.240310    2569 generic.go:356] "Generic (PLEG): container finished" podID="6125c782-f1be-4026-a953-12fa5000e291" containerID="461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0" exitCode=0
Jul 28 10:18:41.240394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:41.240356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerDied","Data":"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"}
Jul 28 10:18:55.319771 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:55.319732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerStarted","Data":"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"}
Jul 28 10:18:57.332237 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:57.332192    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerStarted","Data":"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"}
Jul 28 10:18:59.342166 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:59.342128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerStarted","Data":"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"}
Jul 28 10:18:59.342559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:59.342278    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:18:59.363998 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:18:59.363939    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podStartSLOduration=2.066927889 podStartE2EDuration="23.363924704s" totalImagesPullingTime="17.287855239s" totalInitContainerRuntime="4.009141576s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:18:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:18:59.363109639 +0000 UTC m=+697.283813691" watchObservedRunningTime="2026-07-28 10:18:59.363924704 +0000 UTC m=+697.284628756"
Jul 28 10:19:00.347049 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:00.347016    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:19:00.347415 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:00.347078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:19:00.348286 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:00.348259    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:00.348950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:00.348929    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:01.350637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:01.350588    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:01.351132 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:01.350985    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:01.354235 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:01.354216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:19:02.354465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:02.354428    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:02.354912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:02.354797    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:12.354956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:12.354903    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:12.355424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:12.355315    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:22.354444 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:22.354348    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:22.354900 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:22.354875    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:32.355259 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:32.355208    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:32.355788 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:32.355570    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:42.355353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:42.355295    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:42.355861 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:42.355733    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:19:52.354883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:52.354832    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:19:52.355443 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:19:52.355283    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:02.354935 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:02.354885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:20:02.355427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:02.355370    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:07.561444 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:07.561399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:20:07.561932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:07.561799    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:17.562773 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:17.562711    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:20:17.563157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:17.562874    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:20:31.502563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.502524    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:20:31.503319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.502844    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" containerID="cri-o://9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0" gracePeriod=30
Jul 28 10:20:31.503319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.502899    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" containerID="cri-o://655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33" gracePeriod=30
Jul 28 10:20:31.503319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.502854    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" containerID="cri-o://3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049" gracePeriod=30
Jul 28 10:20:31.525348 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.525319    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:20:31.528278 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.528241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.531099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.531083    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 10:20:31.531173 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.531116    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 10:20:31.540274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.540247    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:20:31.645186 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.645151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.645186 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.645196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.645428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.645223    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.645428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.645262    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7g6\" (UniqueName: \"kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.645428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.645303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.706640 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.706610    2569 generic.go:356] "Generic (PLEG): container finished" podID="6125c782-f1be-4026-a953-12fa5000e291" containerID="655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33" exitCode=2
Jul 28 10:20:31.706869 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.706648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerDied","Data":"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"}
Jul 28 10:20:31.746483 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.746444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.746483 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.746487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.746760 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.746612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.746760 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.746657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7g6\" (UniqueName: \"kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.746760 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.746715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.747074 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.747023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.747265 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.747242    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.747463 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.747446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.748966 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.748946    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.755895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.755844    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7g6\" (UniqueName: \"kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6\") pod \"isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.837932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.837880    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:31.970993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:31.970969    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:20:31.973443 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:20:31.973419    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace465a_fe43_45ba_bb25_cb49ee628d0c.slice/crio-61e4271498960c4a94e00ae9bffa639de91e9636c8693880193b42f8f60c80c3 WatchSource:0}: Error finding container 61e4271498960c4a94e00ae9bffa639de91e9636c8693880193b42f8f60c80c3: Status 404 returned error can't find the container with id 61e4271498960c4a94e00ae9bffa639de91e9636c8693880193b42f8f60c80c3
Jul 28 10:20:32.712244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:32.712207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerStarted","Data":"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"}
Jul 28 10:20:32.712632 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:32.712282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerStarted","Data":"61e4271498960c4a94e00ae9bffa639de91e9636c8693880193b42f8f60c80c3"}
Jul 28 10:20:36.350659 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:36.350615    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:20:36.732804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:36.732766    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerID="3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0" exitCode=0
Jul 28 10:20:36.732991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:36.732838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerDied","Data":"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"}
Jul 28 10:20:36.735435 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:36.735412    2569 generic.go:356] "Generic (PLEG): container finished" podID="6125c782-f1be-4026-a953-12fa5000e291" containerID="9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0" exitCode=0
Jul 28 10:20:36.735522 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:36.735462    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerDied","Data":"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"}
Jul 28 10:20:37.561508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.561463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:20:37.561951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.561748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:37.742138 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerStarted","Data":"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"}
Jul 28 10:20:37.742306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerStarted","Data":"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"}
Jul 28 10:20:37.742306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerStarted","Data":"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"}
Jul 28 10:20:37.742620 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:37.742746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742659    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:37.742746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.742714    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:37.744243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.744217    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:20:37.744828 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.744806    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:37.767366 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:37.767320    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podStartSLOduration=3.071383193 podStartE2EDuration="6.767306948s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695923755s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:37.765756233 +0000 UTC m=+795.686460284" watchObservedRunningTime="2026-07-28 10:20:37.767306948 +0000 UTC m=+795.688011000"
Jul 28 10:20:38.746561 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:38.746518    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:20:38.747065 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:38.747010    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:41.351247 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:41.351195    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:20:43.751064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:43.751036    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:20:43.751580 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:43.751553    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:20:43.751817 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:43.751791    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:46.350983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:46.350886    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:20:46.351353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:46.351042    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:20:47.561371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:47.561322    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:20:47.561799 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:47.561627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:51.350818 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:51.350775    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:20:53.752012 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:53.751960    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:20:53.752482 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:53.752338    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:56.351129 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:56.351083    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:20:57.561668 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:57.561632    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:20:57.562121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:57.561800    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:20:57.562121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:57.562037    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:20:57.562190 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:20:57.562136    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:21:01.351762 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.351679    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:21:01.649165 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.649138    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:21:01.699735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.699702    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvnmg\" (UniqueName: \"kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg\") pod \"6125c782-f1be-4026-a953-12fa5000e291\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") "
Jul 28 10:21:01.699912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.699747    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config\") pod \"6125c782-f1be-4026-a953-12fa5000e291\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") "
Jul 28 10:21:01.699912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.699786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert\") pod \"6125c782-f1be-4026-a953-12fa5000e291\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") "
Jul 28 10:21:01.699912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.699851    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location\") pod \"6125c782-f1be-4026-a953-12fa5000e291\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") "
Jul 28 10:21:01.699912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.699884    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls\") pod \"6125c782-f1be-4026-a953-12fa5000e291\" (UID: \"6125c782-f1be-4026-a953-12fa5000e291\") "
Jul 28 10:21:01.700189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.700161    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6125c782-f1be-4026-a953-12fa5000e291" (UID: "6125c782-f1be-4026-a953-12fa5000e291"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:21:01.700189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.700171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6125c782-f1be-4026-a953-12fa5000e291" (UID: "6125c782-f1be-4026-a953-12fa5000e291"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:21:01.700719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.700675    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6125c782-f1be-4026-a953-12fa5000e291" (UID: "6125c782-f1be-4026-a953-12fa5000e291"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:21:01.702039 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.702018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg" (OuterVolumeSpecName: "kube-api-access-cvnmg") pod "6125c782-f1be-4026-a953-12fa5000e291" (UID: "6125c782-f1be-4026-a953-12fa5000e291"). InnerVolumeSpecName "kube-api-access-cvnmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:21:01.702129 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.702087    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6125c782-f1be-4026-a953-12fa5000e291" (UID: "6125c782-f1be-4026-a953-12fa5000e291"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:21:01.801473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.801440    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6125c782-f1be-4026-a953-12fa5000e291-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:21:01.801473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.801469    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6125c782-f1be-4026-a953-12fa5000e291-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:21:01.801473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.801480    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnmg\" (UniqueName: \"kubernetes.io/projected/6125c782-f1be-4026-a953-12fa5000e291-kube-api-access-cvnmg\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:21:01.801726 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.801489    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:21:01.801726 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.801498    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6125c782-f1be-4026-a953-12fa5000e291-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:21:01.848067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.848037    2569 generic.go:356] "Generic (PLEG): container finished" podID="6125c782-f1be-4026-a953-12fa5000e291" containerID="3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049" exitCode=0
Jul 28 10:21:01.848221 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.848094    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"
Jul 28 10:21:01.848221 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.848094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerDied","Data":"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"}
Jul 28 10:21:01.848296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.848256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s" event={"ID":"6125c782-f1be-4026-a953-12fa5000e291","Type":"ContainerDied","Data":"f6119d0296daf6b349c126be282a4a3d0c8b78f4d0c3eb84b61bc5b805232aef"}
Jul 28 10:21:01.848296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.848291    2569 scope.go:117] "RemoveContainer" containerID="3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"
Jul 28 10:21:01.857178 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.857156    2569 scope.go:117] "RemoveContainer" containerID="655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"
Jul 28 10:21:01.864893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.864871    2569 scope.go:117] "RemoveContainer" containerID="9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"
Jul 28 10:21:01.871821 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.871794    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:21:01.873160 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.873128    2569 scope.go:117] "RemoveContainer" containerID="461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"
Jul 28 10:21:01.875506 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.875486    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6d9fb6f59c-wvf9s"]
Jul 28 10:21:01.881648 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.881624    2569 scope.go:117] "RemoveContainer" containerID="3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"
Jul 28 10:21:01.881973 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:21:01.881932    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049\": container with ID starting with 3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049 not found: ID does not exist" containerID="3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"
Jul 28 10:21:01.882033 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.881964    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049"} err="failed to get container status \"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049\": rpc error: code = NotFound desc = could not find container \"3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049\": container with ID starting with 3b8621f0bc40d859feae60e24beb7bf543cc86d22a2faeb63678a42fbe0c3049 not found: ID does not exist"
Jul 28 10:21:01.882033 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.881985    2569 scope.go:117] "RemoveContainer" containerID="655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"
Jul 28 10:21:01.882338 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:21:01.882323    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33\": container with ID starting with 655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33 not found: ID does not exist" containerID="655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"
Jul 28 10:21:01.882412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.882342    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33"} err="failed to get container status \"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33\": rpc error: code = NotFound desc = could not find container \"655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33\": container with ID starting with 655931579e7711066709c9effaa20e15a79381b74de58c1d83f94bc9c81d0e33 not found: ID does not exist"
Jul 28 10:21:01.882412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.882360    2569 scope.go:117] "RemoveContainer" containerID="9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"
Jul 28 10:21:01.882653 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:21:01.882634    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0\": container with ID starting with 9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0 not found: ID does not exist" containerID="9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"
Jul 28 10:21:01.882797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.882660    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0"} err="failed to get container status \"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0\": rpc error: code = NotFound desc = could not find container \"9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0\": container with ID starting with 9625523990bc07563610c9ea202255f47a9bdb91b0c11866ae835804c788ebe0 not found: ID does not exist"
Jul 28 10:21:01.882797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.882678    2569 scope.go:117] "RemoveContainer" containerID="461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"
Jul 28 10:21:01.883054 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:21:01.883037    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0\": container with ID starting with 461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0 not found: ID does not exist" containerID="461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"
Jul 28 10:21:01.883110 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:01.883061    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0"} err="failed to get container status \"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0\": rpc error: code = NotFound desc = could not find container \"461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0\": container with ID starting with 461c1897e21c366eb5906b4280925c18da7482bd8e796a14188ac50fb632a4f0 not found: ID does not exist"
Jul 28 10:21:02.565505 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:02.565476    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6125c782-f1be-4026-a953-12fa5000e291" path="/var/lib/kubelet/pods/6125c782-f1be-4026-a953-12fa5000e291/volumes"
Jul 28 10:21:03.751594 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:03.751553    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:21:03.752061 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:03.752037    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:21:13.752085 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:13.752029    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:21:13.752566 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:13.752457    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:21:23.751796 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:23.751706    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:21:23.752247 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:23.752057    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:21:33.752239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:33.752192    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:21:33.752740 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:33.752653    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:21:43.752377 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:43.752332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:21:43.752904 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:43.752795    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:21:53.751887 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:53.751850    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:21:53.752285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:21:53.751973    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:06.713834 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.713798    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714124    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714132    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714138    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714147    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714152    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714161    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="storage-initializer"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714166    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="storage-initializer"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714219    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kserve-container"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714232    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="kube-rbac-proxy"
Jul 28 10:22:06.714511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.714241    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6125c782-f1be-4026-a953-12fa5000e291" containerName="agent"
Jul 28 10:22:06.717501 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.717476    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.720537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.720495    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 10:22:06.721330 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.721305    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 10:22:06.731939 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.731914    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:22:06.748448 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.748416    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:22:06.748735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.748706    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" containerID="cri-o://2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb" gracePeriod=30
Jul 28 10:22:06.748891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.748760    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" containerID="cri-o://5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1" gracePeriod=30
Jul 28 10:22:06.748891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.748788    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" containerID="cri-o://869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee" gracePeriod=30
Jul 28 10:22:06.838342 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.838310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7mb\" (UniqueName: \"kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.838503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.838349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.838503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.838385    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.939458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.939407    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7mb\" (UniqueName: \"kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.939458 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.939448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.939729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.939487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.940244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.940213    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.941946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.941924    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:06.947991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:06.947969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7mb\" (UniqueName: \"kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb\") pod \"message-dumper-predictor-6575bcfdfd-gz799\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:07.030178 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:07.030086    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:07.112653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:07.112616    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerID="869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee" exitCode=2
Jul 28 10:22:07.112818 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:07.112677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerDied","Data":"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"}
Jul 28 10:22:07.180881 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:07.180848    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:22:07.184272 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:22:07.184250    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a631c4f_9b9e_4f45_bc07_3ebbb1435c74.slice/crio-1533ece1ae13defc06257ae5a3a1e24231ebd25cce8d0fb55d2c648728b5000b WatchSource:0}: Error finding container 1533ece1ae13defc06257ae5a3a1e24231ebd25cce8d0fb55d2c648728b5000b: Status 404 returned error can't find the container with id 1533ece1ae13defc06257ae5a3a1e24231ebd25cce8d0fb55d2c648728b5000b
Jul 28 10:22:07.186061 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:07.186046    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:22:08.120761 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:08.120724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerStarted","Data":"1533ece1ae13defc06257ae5a3a1e24231ebd25cce8d0fb55d2c648728b5000b"}
Jul 28 10:22:08.747671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:08.747559    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:09.129443 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.129351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerStarted","Data":"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"}
Jul 28 10:22:09.129443 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.129424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerStarted","Data":"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"}
Jul 28 10:22:09.129896 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.129601    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:09.129896 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.129725    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:09.131427 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.131408    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:09.149471 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:09.149418    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" podStartSLOduration=1.9302503930000001 podStartE2EDuration="3.149403232s" totalImagesPullingTime="1.219152839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:09.148412764 +0000 UTC m=+887.069116818" watchObservedRunningTime="2026-07-28 10:22:09.149403232 +0000 UTC m=+887.070107284"
Jul 28 10:22:12.143977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:12.143946    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerID="2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb" exitCode=0
Jul 28 10:22:12.144367 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:12.143993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerDied","Data":"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"}
Jul 28 10:22:13.747193 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:13.747138    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:13.751489 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:13.751456    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:22:13.751741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:13.751719    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:16.142423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.142345    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:22:16.746364 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.746334    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:22:16.750067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.750050    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:16.752647 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.752627    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 10:22:16.752779 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.752711    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 10:22:16.760498 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.760474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:22:16.919112 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.919075    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:16.919112 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.919116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:16.919334 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.919153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:16.919334 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.919283    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7bt\" (UniqueName: \"kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:16.919334 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:16.919333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.019814 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.019716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.019814 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.019761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.019814 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.019792    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.020087 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.019866    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7bt\" (UniqueName: \"kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.020087 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.019908    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.020087 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:17.019946    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 28 10:22:17.020087 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:17.020026    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls podName:6bbb4ff9-c163-4924-b18f-c328bdc725c1 nodeName:}" failed. No retries permitted until 2026-07-28 10:22:17.520006671 +0000 UTC m=+895.440710722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls") pod "isvc-logger-predictor-688776f7db-wfw27" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1") : secret "isvc-logger-predictor-serving-cert" not found
Jul 28 10:22:17.020261 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.020240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.020560 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.020542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.020595 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.020543    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.032157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.032133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7bt\" (UniqueName: \"kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.524242 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.524199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.526576 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.526555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") pod \"isvc-logger-predictor-688776f7db-wfw27\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.661415 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.661377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:17.797764 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:17.797737    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:22:17.799981 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:22:17.799952    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbb4ff9_c163_4924_b18f_c328bdc725c1.slice/crio-f83efbbeff76db0f46a16a13671e18d0fa70f5c43a1ab9feaac0ad7f11f91002 WatchSource:0}: Error finding container f83efbbeff76db0f46a16a13671e18d0fa70f5c43a1ab9feaac0ad7f11f91002: Status 404 returned error can't find the container with id f83efbbeff76db0f46a16a13671e18d0fa70f5c43a1ab9feaac0ad7f11f91002
Jul 28 10:22:18.171599 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:18.171559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerStarted","Data":"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"}
Jul 28 10:22:18.171789 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:18.171626    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerStarted","Data":"f83efbbeff76db0f46a16a13671e18d0fa70f5c43a1ab9feaac0ad7f11f91002"}
Jul 28 10:22:18.747541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:18.747499    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:18.748016 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:18.747656    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:22.189598 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:22.189562    2569 generic.go:356] "Generic (PLEG): container finished" podID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerID="716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd" exitCode=0
Jul 28 10:22:22.190106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:22.189644    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerDied","Data":"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"}
Jul 28 10:22:22.451075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:22.451006    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:22:22.451231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:22.451155    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:22:23.197640 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.197601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerStarted","Data":"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"}
Jul 28 10:22:23.198128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.197681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerStarted","Data":"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"}
Jul 28 10:22:23.198128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.197732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerStarted","Data":"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"}
Jul 28 10:22:23.198128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.197979    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:23.220939 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.220886    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podStartSLOduration=3.506001582 podStartE2EDuration="7.220871658s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714870076s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:22:23.218601611 +0000 UTC m=+901.139305664" watchObservedRunningTime="2026-07-28 10:22:23.220871658 +0000 UTC m=+901.141575710"
Jul 28 10:22:23.747792 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.747746    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:23.752101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.752070    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:22:23.752397 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:23.752374    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:24.203418 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:24.203385    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:24.203869 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:24.203462    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:24.205087 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:24.205051    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:24.205679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:24.205655    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:25.207838 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:25.207797    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:25.208250 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:25.208164    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:28.747511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:28.747463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:30.212015 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:30.211984    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:22:30.212623 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:30.212583    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:30.212957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:30.212937    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:33.747261 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:33.747218    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 10:22:33.752004 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:33.751975    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:5000: connect: connection refused"
Jul 28 10:22:33.752186 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:33.752168    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:33.752346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:33.752329    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:33.752409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:33.752399    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:36.893589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.893557    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:36.990919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.990880    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7g6\" (UniqueName: \"kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6\") pod \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") "
Jul 28 10:22:36.990919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.990918    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert\") pod \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") "
Jul 28 10:22:36.991192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.990960    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config\") pod \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") "
Jul 28 10:22:36.991192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.991005    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls\") pod \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") "
Jul 28 10:22:36.991192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.991025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location\") pod \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\" (UID: \"1ace465a-fe43-45ba-bb25-cb49ee628d0c\") "
Jul 28 10:22:36.991473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.991439    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ace465a-fe43-45ba-bb25-cb49ee628d0c" (UID: "1ace465a-fe43-45ba-bb25-cb49ee628d0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:22:36.991567 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.991469    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ace465a-fe43-45ba-bb25-cb49ee628d0c" (UID: "1ace465a-fe43-45ba-bb25-cb49ee628d0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:36.991567 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.991468    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ace465a-fe43-45ba-bb25-cb49ee628d0c" (UID: "1ace465a-fe43-45ba-bb25-cb49ee628d0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:22:36.993116 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.993091    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ace465a-fe43-45ba-bb25-cb49ee628d0c" (UID: "1ace465a-fe43-45ba-bb25-cb49ee628d0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:22:36.993215 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:36.993153    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6" (OuterVolumeSpecName: "kube-api-access-5p7g6") pod "1ace465a-fe43-45ba-bb25-cb49ee628d0c" (UID: "1ace465a-fe43-45ba-bb25-cb49ee628d0c"). InnerVolumeSpecName "kube-api-access-5p7g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:22:37.092101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.092007    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:22:37.092101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.092037    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ace465a-fe43-45ba-bb25-cb49ee628d0c-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:22:37.092101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.092051    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:22:37.092101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.092063    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7g6\" (UniqueName: \"kubernetes.io/projected/1ace465a-fe43-45ba-bb25-cb49ee628d0c-kube-api-access-5p7g6\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:22:37.092101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.092076    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ace465a-fe43-45ba-bb25-cb49ee628d0c-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:22:37.261076 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.261038    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerID="5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1" exitCode=0
Jul 28 10:22:37.261245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.261107    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"
Jul 28 10:22:37.261245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.261118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerDied","Data":"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"}
Jul 28 10:22:37.261245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.261197    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f" event={"ID":"1ace465a-fe43-45ba-bb25-cb49ee628d0c","Type":"ContainerDied","Data":"61e4271498960c4a94e00ae9bffa639de91e9636c8693880193b42f8f60c80c3"}
Jul 28 10:22:37.261245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.261241    2569 scope.go:117] "RemoveContainer" containerID="5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"
Jul 28 10:22:37.269481 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.269462    2569 scope.go:117] "RemoveContainer" containerID="869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"
Jul 28 10:22:37.276537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.276521    2569 scope.go:117] "RemoveContainer" containerID="2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"
Jul 28 10:22:37.283284 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.283264    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:22:37.284111 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.284100    2569 scope.go:117] "RemoveContainer" containerID="3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"
Jul 28 10:22:37.286943 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.286923    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-67bdbdb944-z5k7f"]
Jul 28 10:22:37.291459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.291437    2569 scope.go:117] "RemoveContainer" containerID="5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"
Jul 28 10:22:37.291747 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:37.291731    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1\": container with ID starting with 5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1 not found: ID does not exist" containerID="5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"
Jul 28 10:22:37.291806 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.291755    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1"} err="failed to get container status \"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1\": rpc error: code = NotFound desc = could not find container \"5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1\": container with ID starting with 5860b59307df5b63b3482af2b6682176d4ca18f49c5da0e52da49e4ec0f09ac1 not found: ID does not exist"
Jul 28 10:22:37.291806 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.291774    2569 scope.go:117] "RemoveContainer" containerID="869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"
Jul 28 10:22:37.292023 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:37.292008    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee\": container with ID starting with 869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee not found: ID does not exist" containerID="869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"
Jul 28 10:22:37.292064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.292025    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee"} err="failed to get container status \"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee\": rpc error: code = NotFound desc = could not find container \"869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee\": container with ID starting with 869c18906ab4b861de5cee396e52ece6a5dce035d46d644493caf3e571c416ee not found: ID does not exist"
Jul 28 10:22:37.292064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.292038    2569 scope.go:117] "RemoveContainer" containerID="2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"
Jul 28 10:22:37.292266 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:37.292250    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb\": container with ID starting with 2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb not found: ID does not exist" containerID="2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"
Jul 28 10:22:37.292314 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.292269    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb"} err="failed to get container status \"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb\": rpc error: code = NotFound desc = could not find container \"2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb\": container with ID starting with 2d96c834b1b55fe7cd0dad0884132b0d6b9c883222d2e89f5126eff5899ecaeb not found: ID does not exist"
Jul 28 10:22:37.292314 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.292281    2569 scope.go:117] "RemoveContainer" containerID="3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"
Jul 28 10:22:37.292542 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:22:37.292516    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0\": container with ID starting with 3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0 not found: ID does not exist" containerID="3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"
Jul 28 10:22:37.292632 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:37.292551    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0"} err="failed to get container status \"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0\": rpc error: code = NotFound desc = could not find container \"3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0\": container with ID starting with 3bc6fd180c145a77689bdf8e85050b591fe9c127f46dff1a4c6c02b21c8abaa0 not found: ID does not exist"
Jul 28 10:22:38.565707 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:38.565662    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" path="/var/lib/kubelet/pods/1ace465a-fe43-45ba-bb25-cb49ee628d0c/volumes"
Jul 28 10:22:40.212930 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:40.212887    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:40.213423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:40.213397    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:22:50.213239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:50.213196    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:22:50.213778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:22:50.213711    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:00.212811 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:00.212766    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:00.213213 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:00.213184    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:10.213423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:10.213382    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:10.213957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:10.213935    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:20.213009 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:20.212963    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:20.213452 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:20.213432    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:30.213250 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:30.213207    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:30.213731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:30.213710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:33.561434 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:33.561388    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:33.561878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:33.561658    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:43.562659 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:43.562630    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:23:43.563282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:43.562826    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:23:51.736540 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:51.736461    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-gz799_3a631c4f-9b9e-4f45-bc07-3ebbb1435c74/kserve-container/0.log"
Jul 28 10:23:51.964153 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:51.964120    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:23:51.964465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:51.964424    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" containerID="cri-o://84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5" gracePeriod=30
Jul 28 10:23:51.964637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:51.964465    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" containerID="cri-o://d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6" gracePeriod=30
Jul 28 10:23:51.964637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:51.964470    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" containerID="cri-o://898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4" gracePeriod=30
Jul 28 10:23:52.044397 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044300    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:23:52.044660 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044646    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044662    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044673    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044679    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044703    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="storage-initializer"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044713    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="storage-initializer"
Jul 28 10:23:52.044729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044731    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy"
Jul 28 10:23:52.044911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044737    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy"
Jul 28 10:23:52.044911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044789    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="agent"
Jul 28 10:23:52.044911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044796    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kserve-container"
Jul 28 10:23:52.044911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.044803    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace465a-fe43-45ba-bb25-cb49ee628d0c" containerName="kube-rbac-proxy"
Jul 28 10:23:52.047911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.047893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.051001 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.050979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 10:23:52.051128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.051023    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 10:23:52.059736 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.059708    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:23:52.095292 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.095255    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:23:52.095568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.095524    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kserve-container" containerID="cri-o://f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce" gracePeriod=30
Jul 28 10:23:52.095658 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.095556    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kube-rbac-proxy" containerID="cri-o://8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c" gracePeriod=30
Jul 28 10:23:52.130910 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.130876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.131092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.130921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.131092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.130957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.131092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.130981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfq56\" (UniqueName: \"kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.131092 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.131025    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232305 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232490 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232473    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfq56\" (UniqueName: \"kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.232762 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.232742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.233167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.233135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.233268 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.233135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.234703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.234657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.242143 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.242119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfq56\" (UniqueName: \"kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cgdd7\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.336122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.336096    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:23:52.358034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.358000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:23:52.433928 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.433897    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk7mb\" (UniqueName: \"kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb\") pod \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") "
Jul 28 10:23:52.434078 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.433958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config\") pod \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") "
Jul 28 10:23:52.434078 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.434007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls\") pod \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\" (UID: \"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74\") "
Jul 28 10:23:52.434386 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.434357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" (UID: "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:23:52.436028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.435999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb" (OuterVolumeSpecName: "kube-api-access-sk7mb") pod "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" (UID: "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74"). InnerVolumeSpecName "kube-api-access-sk7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:23:52.436139 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.436040    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" (UID: "3a631c4f-9b9e-4f45-bc07-3ebbb1435c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:23:52.493844 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.493817    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:23:52.496147 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:23:52.496111    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324c6db2_a09d_4b33_84d2_52eba2f5273a.slice/crio-241340e7110c725b6168c0c05b629eb233c763fcf484c2ebf9954b807f8b34b1 WatchSource:0}: Error finding container 241340e7110c725b6168c0c05b629eb233c763fcf484c2ebf9954b807f8b34b1: Status 404 returned error can't find the container with id 241340e7110c725b6168c0c05b629eb233c763fcf484c2ebf9954b807f8b34b1
Jul 28 10:23:52.535344 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.535321    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:23:52.535344 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.535343    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7mb\" (UniqueName: \"kubernetes.io/projected/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-api-access-sk7mb\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:23:52.535547 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.535354    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:23:52.580198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580115    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerID="8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c" exitCode=2
Jul 28 10:23:52.580198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580141    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerID="f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce" exitCode=2
Jul 28 10:23:52.580198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580175    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"
Jul 28 10:23:52.580460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerDied","Data":"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"}
Jul 28 10:23:52.580460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerDied","Data":"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"}
Jul 28 10:23:52.580460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799" event={"ID":"3a631c4f-9b9e-4f45-bc07-3ebbb1435c74","Type":"ContainerDied","Data":"1533ece1ae13defc06257ae5a3a1e24231ebd25cce8d0fb55d2c648728b5000b"}
Jul 28 10:23:52.580460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.580326    2569 scope.go:117] "RemoveContainer" containerID="8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"
Jul 28 10:23:52.583240 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.583214    2569 generic.go:356] "Generic (PLEG): container finished" podID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerID="d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6" exitCode=2
Jul 28 10:23:52.583361 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.583290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerDied","Data":"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"}
Jul 28 10:23:52.585025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.584999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerStarted","Data":"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"}
Jul 28 10:23:52.585124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.585066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerStarted","Data":"241340e7110c725b6168c0c05b629eb233c763fcf484c2ebf9954b807f8b34b1"}
Jul 28 10:23:52.589027 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.589005    2569 scope.go:117] "RemoveContainer" containerID="f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"
Jul 28 10:23:52.597417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.597393    2569 scope.go:117] "RemoveContainer" containerID="8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"
Jul 28 10:23:52.597772 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:23:52.597742    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c\": container with ID starting with 8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c not found: ID does not exist" containerID="8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"
Jul 28 10:23:52.597859 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.597782    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"} err="failed to get container status \"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c\": rpc error: code = NotFound desc = could not find container \"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c\": container with ID starting with 8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c not found: ID does not exist"
Jul 28 10:23:52.597859 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.597807    2569 scope.go:117] "RemoveContainer" containerID="f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"
Jul 28 10:23:52.598082 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:23:52.598062    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce\": container with ID starting with f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce not found: ID does not exist" containerID="f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"
Jul 28 10:23:52.598128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598090    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"} err="failed to get container status \"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce\": rpc error: code = NotFound desc = could not find container \"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce\": container with ID starting with f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce not found: ID does not exist"
Jul 28 10:23:52.598128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598109    2569 scope.go:117] "RemoveContainer" containerID="8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"
Jul 28 10:23:52.598383 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598331    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c"} err="failed to get container status \"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c\": rpc error: code = NotFound desc = could not find container \"8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c\": container with ID starting with 8ecc3860da40f91d93c048741ad13a3bdc343b9b572441b640be22f87c17114c not found: ID does not exist"
Jul 28 10:23:52.598478 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598384    2569 scope.go:117] "RemoveContainer" containerID="f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"
Jul 28 10:23:52.598527 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598494    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:23:52.598654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.598631    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce"} err="failed to get container status \"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce\": rpc error: code = NotFound desc = could not find container \"f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce\": container with ID starting with f268efa8502fda51bee0b074af52c5f29631f2dddd1e53c4c9fbdc35356df7ce not found: ID does not exist"
Jul 28 10:23:52.601072 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:52.601053    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gz799"]
Jul 28 10:23:53.562001 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:53.561955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:23:53.562419 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:53.562359    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:23:54.565965 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:54.565935    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" path="/var/lib/kubelet/pods/3a631c4f-9b9e-4f45-bc07-3ebbb1435c74/volumes"
Jul 28 10:23:55.208191 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:55.208150    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:23:57.608339 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:57.608305    2569 generic.go:356] "Generic (PLEG): container finished" podID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerID="84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5" exitCode=0
Jul 28 10:23:57.608740 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:23:57.608371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerDied","Data":"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"}
Jul 28 10:24:00.209039 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:00.208995    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:24:03.562143 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:03.562098    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:24:03.562572 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:03.562480    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:05.208356 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:05.208311    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:24:05.208753 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:05.208446    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:24:10.208568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:10.208519    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:24:13.562031 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:13.561987    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:24:13.562496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:13.562150    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:24:13.562496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:13.562336    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 10:24:13.562496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:13.562427    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:24:15.208182 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:15.208137    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:24:20.208795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:20.208747    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:24:22.122933 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.122896    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:24:22.192408 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192373    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config\") pod \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") "
Jul 28 10:24:22.192408 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192412    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location\") pod \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") "
Jul 28 10:24:22.192642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192455    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd7bt\" (UniqueName: \"kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt\") pod \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") "
Jul 28 10:24:22.192642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192474    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") pod \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") "
Jul 28 10:24:22.192642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192500    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert\") pod \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\" (UID: \"6bbb4ff9-c163-4924-b18f-c328bdc725c1\") "
Jul 28 10:24:22.192878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192847    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6bbb4ff9-c163-4924-b18f-c328bdc725c1" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:24:22.192931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bbb4ff9-c163-4924-b18f-c328bdc725c1" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:24:22.192970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.192944    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6bbb4ff9-c163-4924-b18f-c328bdc725c1" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:24:22.194594 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.194574    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bbb4ff9-c163-4924-b18f-c328bdc725c1" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:24:22.194657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.194631    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt" (OuterVolumeSpecName: "kube-api-access-pd7bt") pod "6bbb4ff9-c163-4924-b18f-c328bdc725c1" (UID: "6bbb4ff9-c163-4924-b18f-c328bdc725c1"). InnerVolumeSpecName "kube-api-access-pd7bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:24:22.294050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.293973    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:24:22.294050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.293999    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:24:22.294050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.294009    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7bt\" (UniqueName: \"kubernetes.io/projected/6bbb4ff9-c163-4924-b18f-c328bdc725c1-kube-api-access-pd7bt\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:24:22.294050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.294018    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bbb4ff9-c163-4924-b18f-c328bdc725c1-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:24:22.294050 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.294028    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bbb4ff9-c163-4924-b18f-c328bdc725c1-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:24:22.718147 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.718112    2569 generic.go:356] "Generic (PLEG): container finished" podID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerID="898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4" exitCode=137
Jul 28 10:24:22.718319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.718179    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"
Jul 28 10:24:22.718319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.718174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerDied","Data":"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"}
Jul 28 10:24:22.718425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.718353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27" event={"ID":"6bbb4ff9-c163-4924-b18f-c328bdc725c1","Type":"ContainerDied","Data":"f83efbbeff76db0f46a16a13671e18d0fa70f5c43a1ab9feaac0ad7f11f91002"}
Jul 28 10:24:22.718425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.718375    2569 scope.go:117] "RemoveContainer" containerID="898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"
Jul 28 10:24:22.720002 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.719976    2569 generic.go:356] "Generic (PLEG): container finished" podID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerID="1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047" exitCode=0
Jul 28 10:24:22.720137 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.720035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerDied","Data":"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"}
Jul 28 10:24:22.727307 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.727281    2569 scope.go:117] "RemoveContainer" containerID="d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"
Jul 28 10:24:22.735005 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.734988    2569 scope.go:117] "RemoveContainer" containerID="84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"
Jul 28 10:24:22.742072 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.742052    2569 scope.go:117] "RemoveContainer" containerID="716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"
Jul 28 10:24:22.748879 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.748858    2569 scope.go:117] "RemoveContainer" containerID="898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"
Jul 28 10:24:22.749125 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:24:22.749107    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4\": container with ID starting with 898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4 not found: ID does not exist" containerID="898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"
Jul 28 10:24:22.749204 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749137    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4"} err="failed to get container status \"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4\": rpc error: code = NotFound desc = could not find container \"898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4\": container with ID starting with 898d78c0e12645e568b1e62e21633fc09e4d37d9c05221329ba2f7e7a3610ac4 not found: ID does not exist"
Jul 28 10:24:22.749204 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749164    2569 scope.go:117] "RemoveContainer" containerID="d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"
Jul 28 10:24:22.749390 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:24:22.749374    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6\": container with ID starting with d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6 not found: ID does not exist" containerID="d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"
Jul 28 10:24:22.749432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749396    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6"} err="failed to get container status \"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6\": rpc error: code = NotFound desc = could not find container \"d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6\": container with ID starting with d8904a8fb70a3ccd56e68eb0ca3c61ccc5c1ebe7fb3803ae44fe98f60e3422e6 not found: ID does not exist"
Jul 28 10:24:22.749432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749410    2569 scope.go:117] "RemoveContainer" containerID="84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"
Jul 28 10:24:22.749625 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:24:22.749609    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5\": container with ID starting with 84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5 not found: ID does not exist" containerID="84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"
Jul 28 10:24:22.749703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749628    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5"} err="failed to get container status \"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5\": rpc error: code = NotFound desc = could not find container \"84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5\": container with ID starting with 84fa9d9babc7c9e1e75a3aeba135fd2923a4d6b6ce9a343e997eaba86eb7fab5 not found: ID does not exist"
Jul 28 10:24:22.749703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749641    2569 scope.go:117] "RemoveContainer" containerID="716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"
Jul 28 10:24:22.749898 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:24:22.749884    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd\": container with ID starting with 716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd not found: ID does not exist" containerID="716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"
Jul 28 10:24:22.749941 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.749901    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd"} err="failed to get container status \"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd\": rpc error: code = NotFound desc = could not find container \"716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd\": container with ID starting with 716fc695ff54e300154e265f7984c68a42524443a27870655a91fbcca0221dcd not found: ID does not exist"
Jul 28 10:24:22.753744 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.753717    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:24:22.755544 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:22.755524    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-688776f7db-wfw27"]
Jul 28 10:24:24.567542 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:24:24.567511    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" path="/var/lib/kubelet/pods/6bbb4ff9-c163-4924-b18f-c328bdc725c1/volumes"
Jul 28 10:26:28.289932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:28.289892    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerStarted","Data":"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"}
Jul 28 10:26:28.290454 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:28.289993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerStarted","Data":"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"}
Jul 28 10:26:28.290454 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:28.290065    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:26:28.317518 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:28.317452    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podStartSLOduration=1.909462511 podStartE2EDuration="2m36.317434489s" totalImagesPullingTime="2m4.834733291s" totalInitContainerRuntime="29.573238687s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:26:28.316538643 +0000 UTC m=+1146.237242692" watchObservedRunningTime="2026-07-28 10:26:28.317434489 +0000 UTC m=+1146.238138543"
Jul 28 10:26:29.293971 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:29.293933    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:26:29.295352 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:29.295321    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:26:30.298341 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:30.298302    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:26:35.302613 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:35.302583    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:26:35.303189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:35.303126    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:26:45.303892 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:45.303812    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:26:58.783432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:58.783393    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:26:58.783973 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:58.783707    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container" containerID="cri-o://578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df" gracePeriod=30
Jul 28 10:26:58.783973 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:58.783753    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kube-rbac-proxy" containerID="cri-o://5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f" gracePeriod=30
Jul 28 10:26:59.423246 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:59.423211    2569 generic.go:356] "Generic (PLEG): container finished" podID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerID="5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f" exitCode=2
Jul 28 10:26:59.423416 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:26:59.423263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerDied","Data":"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"}
Jul 28 10:27:00.362753 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.362722    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:27:00.389914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.389850    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location\") pod \"324c6db2-a09d-4b33-84d2-52eba2f5273a\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") "
Jul 28 10:27:00.389914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.389889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls\") pod \"324c6db2-a09d-4b33-84d2-52eba2f5273a\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") "
Jul 28 10:27:00.389914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.389914    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfq56\" (UniqueName: \"kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56\") pod \"324c6db2-a09d-4b33-84d2-52eba2f5273a\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") "
Jul 28 10:27:00.390151 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.389979    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert\") pod \"324c6db2-a09d-4b33-84d2-52eba2f5273a\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") "
Jul 28 10:27:00.390151 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.390035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config\") pod \"324c6db2-a09d-4b33-84d2-52eba2f5273a\" (UID: \"324c6db2-a09d-4b33-84d2-52eba2f5273a\") "
Jul 28 10:27:00.390434 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.390398    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "324c6db2-a09d-4b33-84d2-52eba2f5273a" (UID: "324c6db2-a09d-4b33-84d2-52eba2f5273a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:27:00.390570 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.390490    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "324c6db2-a09d-4b33-84d2-52eba2f5273a" (UID: "324c6db2-a09d-4b33-84d2-52eba2f5273a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:27:00.392024 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.391991    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "324c6db2-a09d-4b33-84d2-52eba2f5273a" (UID: "324c6db2-a09d-4b33-84d2-52eba2f5273a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:27:00.392146 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.392125    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56" (OuterVolumeSpecName: "kube-api-access-xfq56") pod "324c6db2-a09d-4b33-84d2-52eba2f5273a" (UID: "324c6db2-a09d-4b33-84d2-52eba2f5273a"). InnerVolumeSpecName "kube-api-access-xfq56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:27:00.393388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.393367    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "324c6db2-a09d-4b33-84d2-52eba2f5273a" (UID: "324c6db2-a09d-4b33-84d2-52eba2f5273a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:27:00.429758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.429727    2569 generic.go:356] "Generic (PLEG): container finished" podID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerID="578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df" exitCode=0
Jul 28 10:27:00.429950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.429788    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"
Jul 28 10:27:00.429950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.429803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerDied","Data":"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"}
Jul 28 10:27:00.429950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.429912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" event={"ID":"324c6db2-a09d-4b33-84d2-52eba2f5273a","Type":"ContainerDied","Data":"241340e7110c725b6168c0c05b629eb233c763fcf484c2ebf9954b807f8b34b1"}
Jul 28 10:27:00.429950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.429950    2569 scope.go:117] "RemoveContainer" containerID="5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"
Jul 28 10:27:00.439257 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.439241    2569 scope.go:117] "RemoveContainer" containerID="578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"
Jul 28 10:27:00.446445 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.446429    2569 scope.go:117] "RemoveContainer" containerID="1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"
Jul 28 10:27:00.453895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.453876    2569 scope.go:117] "RemoveContainer" containerID="5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"
Jul 28 10:27:00.454183 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:27:00.454163    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f\": container with ID starting with 5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f not found: ID does not exist" containerID="5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"
Jul 28 10:27:00.454271 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454192    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f"} err="failed to get container status \"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f\": rpc error: code = NotFound desc = could not find container \"5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f\": container with ID starting with 5841caf750f75c115592ac6222042a4ca46ab81029ea28ec0a1ba76d5661b89f not found: ID does not exist"
Jul 28 10:27:00.454271 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454210    2569 scope.go:117] "RemoveContainer" containerID="578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"
Jul 28 10:27:00.454456 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:27:00.454441    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df\": container with ID starting with 578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df not found: ID does not exist" containerID="578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"
Jul 28 10:27:00.454497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454460    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df"} err="failed to get container status \"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df\": rpc error: code = NotFound desc = could not find container \"578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df\": container with ID starting with 578fe1c00be3ee4363956b556ec94b8d4fdfb67ed78b26fc15d885097e82e7df not found: ID does not exist"
Jul 28 10:27:00.454497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454473    2569 scope.go:117] "RemoveContainer" containerID="1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"
Jul 28 10:27:00.454595 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454510    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:27:00.454732 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:27:00.454709    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047\": container with ID starting with 1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047 not found: ID does not exist" containerID="1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"
Jul 28 10:27:00.454829 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.454734    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047"} err="failed to get container status \"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047\": rpc error: code = NotFound desc = could not find container \"1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047\": container with ID starting with 1e27f0ddfb764b2f267966dca232c756fbcb43958082195c13e742277860d047 not found: ID does not exist"
Jul 28 10:27:00.459895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.459873    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7"]
Jul 28 10:27:00.490668 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.490632    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/324c6db2-a09d-4b33-84d2-52eba2f5273a-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:27:00.490668 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.490666    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/324c6db2-a09d-4b33-84d2-52eba2f5273a-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:27:00.490868 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.490679    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfq56\" (UniqueName: \"kubernetes.io/projected/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-api-access-xfq56\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:27:00.490868 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.490711    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:27:00.490868 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.490720    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/324c6db2-a09d-4b33-84d2-52eba2f5273a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:27:00.566420 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:00.566389    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" path="/var/lib/kubelet/pods/324c6db2-a09d-4b33-84d2-52eba2f5273a/volumes"
Jul 28 10:27:01.300124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:01.300078    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cgdd7" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:27:03.932488 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932452    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932775    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932787    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932795    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932800    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932811    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932816    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932824    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932829    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932834    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="storage-initializer"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932839    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="storage-initializer"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932847    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="storage-initializer"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932852    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="storage-initializer"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932858    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932864    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932879    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932883    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932888    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent"
Jul 28 10:27:03.932931 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932894    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932947    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kube-rbac-proxy"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932957    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a631c4f-9b9e-4f45-bc07-3ebbb1435c74" containerName="kserve-container"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932964    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kserve-container"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932970    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kube-rbac-proxy"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932975    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="agent"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932980    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbb4ff9-c163-4924-b18f-c328bdc725c1" containerName="kserve-container"
Jul 28 10:27:03.933511 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.932984    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="324c6db2-a09d-4b33-84d2-52eba2f5273a" containerName="kube-rbac-proxy"
Jul 28 10:27:03.955106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.955076    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:27:03.955254 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.955199    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:03.957991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.957965    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958008    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958031    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958062    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958119    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958123    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:27:03.958171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:03.958170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:27:04.020795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.020757    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6fp\" (UniqueName: \"kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.020977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.020835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.020977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.020858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.020977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.020880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.020977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.020895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6fp\" (UniqueName: \"kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122382    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.122589 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:27:04.122455    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 10:27:04.122589 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:27:04.122527    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls podName:a1806c31-60eb-4c48-a4ad-b0f2a58d7982 nodeName:}" failed. No retries permitted until 2026-07-28 10:27:04.622507893 +0000 UTC m=+1182.543211923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 28 10:27:04.122905 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.122884    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.123141 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.123125    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.123181 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.123133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.131068 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.131047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6fp\" (UniqueName: \"kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.628088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.628032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.630421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.630388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-rdz2s\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.865969 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.865927    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:04.999423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:04.999372    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:27:05.001753 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:27:05.001724    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1806c31_60eb_4c48_a4ad_b0f2a58d7982.slice/crio-087a84b126c728e68a0f80a4c390de856b2e15831775010fc986ff5c658a0fec WatchSource:0}: Error finding container 087a84b126c728e68a0f80a4c390de856b2e15831775010fc986ff5c658a0fec: Status 404 returned error can't find the container with id 087a84b126c728e68a0f80a4c390de856b2e15831775010fc986ff5c658a0fec
Jul 28 10:27:05.455194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:05.455155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerStarted","Data":"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"}
Jul 28 10:27:05.455346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:05.455218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerStarted","Data":"087a84b126c728e68a0f80a4c390de856b2e15831775010fc986ff5c658a0fec"}
Jul 28 10:27:22.482592 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:22.482562    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:27:22.483061 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:22.482902    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:27:34.576438 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:34.576404    2569 generic.go:356] "Generic (PLEG): container finished" podID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerID="09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13" exitCode=0
Jul 28 10:27:34.576868 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:34.576460    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerDied","Data":"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"}
Jul 28 10:27:35.583088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:35.583053    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerStarted","Data":"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"}
Jul 28 10:27:35.583601 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:35.583140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerStarted","Data":"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"}
Jul 28 10:27:35.583601 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:35.583453    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:35.583601 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:35.583516    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:27:35.603550 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:35.603492    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" podStartSLOduration=3.675388735 podStartE2EDuration="32.603477767s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.928089032s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:27:35.601745628 +0000 UTC m=+1213.522449679" watchObservedRunningTime="2026-07-28 10:27:35.603477767 +0000 UTC m=+1213.524181819"
Jul 28 10:27:41.592992 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:27:41.592961    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:29:08.566433 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:08.566401    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:29:19.601885 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:19.601840    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.29:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:29:20.333533 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:20.333500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:29:20.333846 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:20.333800    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kserve-container" containerID="cri-o://57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148" gracePeriod=30
Jul 28 10:29:20.333922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:20.333838    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kube-rbac-proxy" containerID="cri-o://e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e" gracePeriod=30
Jul 28 10:29:21.013624 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.013591    2569 generic.go:356] "Generic (PLEG): container finished" podID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerID="e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e" exitCode=2
Jul 28 10:29:21.014021 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.013664    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerDied","Data":"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"}
Jul 28 10:29:21.588506 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.588461    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 10:29:21.777657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.777632    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:29:21.857914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.857826    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config\") pod \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") "
Jul 28 10:29:21.857914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.857872    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") pod \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") "
Jul 28 10:29:21.857914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.857895    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np6fp\" (UniqueName: \"kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp\") pod \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") "
Jul 28 10:29:21.857914 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.857913    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert\") pod \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") "
Jul 28 10:29:21.858176 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.857936    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location\") pod \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\" (UID: \"a1806c31-60eb-4c48-a4ad-b0f2a58d7982\") "
Jul 28 10:29:21.858334 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.858302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1806c31-60eb-4c48-a4ad-b0f2a58d7982" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:29:21.858401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.858317    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1806c31-60eb-4c48-a4ad-b0f2a58d7982" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:29:21.860088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.860055    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp" (OuterVolumeSpecName: "kube-api-access-np6fp") pod "a1806c31-60eb-4c48-a4ad-b0f2a58d7982" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982"). InnerVolumeSpecName "kube-api-access-np6fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:29:21.860233 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.860208    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1806c31-60eb-4c48-a4ad-b0f2a58d7982" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:29:21.862029 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.862007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1806c31-60eb-4c48-a4ad-b0f2a58d7982" (UID: "a1806c31-60eb-4c48-a4ad-b0f2a58d7982"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:29:21.959189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.959152    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:29:21.959189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.959183    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:29:21.959189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.959192    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np6fp\" (UniqueName: \"kubernetes.io/projected/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kube-api-access-np6fp\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:29:21.959189 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.959202    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:29:21.959462 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:21.959213    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1806c31-60eb-4c48-a4ad-b0f2a58d7982-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:29:22.019881 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.019847    2569 generic.go:356] "Generic (PLEG): container finished" podID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerID="57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148" exitCode=0
Jul 28 10:29:22.020294 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.019895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerDied","Data":"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"}
Jul 28 10:29:22.020294 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.019908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"
Jul 28 10:29:22.020294 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.019942    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s" event={"ID":"a1806c31-60eb-4c48-a4ad-b0f2a58d7982","Type":"ContainerDied","Data":"087a84b126c728e68a0f80a4c390de856b2e15831775010fc986ff5c658a0fec"}
Jul 28 10:29:22.020294 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.019995    2569 scope.go:117] "RemoveContainer" containerID="e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"
Jul 28 10:29:22.029096 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.029079    2569 scope.go:117] "RemoveContainer" containerID="57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"
Jul 28 10:29:22.036296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.036277    2569 scope.go:117] "RemoveContainer" containerID="09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"
Jul 28 10:29:22.042411 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.042388    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:29:22.043884 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.043868    2569 scope.go:117] "RemoveContainer" containerID="e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"
Jul 28 10:29:22.044171 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:29:22.044139    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e\": container with ID starting with e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e not found: ID does not exist" containerID="e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"
Jul 28 10:29:22.044243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.044185    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e"} err="failed to get container status \"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e\": rpc error: code = NotFound desc = could not find container \"e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e\": container with ID starting with e21090199e32631dca4b747800a67ac6ae69365e05a6850d15d3e48b53c44b9e not found: ID does not exist"
Jul 28 10:29:22.044243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.044212    2569 scope.go:117] "RemoveContainer" containerID="57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"
Jul 28 10:29:22.044509 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:29:22.044491    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148\": container with ID starting with 57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148 not found: ID does not exist" containerID="57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"
Jul 28 10:29:22.044590 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.044514    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148"} err="failed to get container status \"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148\": rpc error: code = NotFound desc = could not find container \"57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148\": container with ID starting with 57295da85eeef1ddc590bed1dfc14cee05685e7322a1a206128c60395823e148 not found: ID does not exist"
Jul 28 10:29:22.044590 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.044532    2569 scope.go:117] "RemoveContainer" containerID="09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"
Jul 28 10:29:22.044813 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:29:22.044792    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13\": container with ID starting with 09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13 not found: ID does not exist" containerID="09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"
Jul 28 10:29:22.044957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.044908    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13"} err="failed to get container status \"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13\": rpc error: code = NotFound desc = could not find container \"09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13\": container with ID starting with 09ecb2e7c8867968478c8b6d298bacd2da94f681948ed2f55613c729afff9d13 not found: ID does not exist"
Jul 28 10:29:22.046212 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.046195    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-rdz2s"]
Jul 28 10:29:22.566214 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:22.566183    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" path="/var/lib/kubelet/pods/a1806c31-60eb-4c48-a4ad-b0f2a58d7982/volumes"
Jul 28 10:29:25.512180 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512140    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512434    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="storage-initializer"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512444    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="storage-initializer"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512457    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kserve-container"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512462    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kserve-container"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512470    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kube-rbac-proxy"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512476    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kube-rbac-proxy"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512524    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kube-rbac-proxy"
Jul 28 10:29:25.512568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.512533    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1806c31-60eb-4c48-a4ad-b0f2a58d7982" containerName="kserve-container"
Jul 28 10:29:25.516859 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.516840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.519318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 10:29:25.519406 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519337    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:29:25.519758 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519745    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:29:25.519841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 10:29:25.519841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519802    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:29:25.519841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.519809    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:29:25.520369 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.520356    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:29:25.525360 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.525338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:29:25.592279 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.592241    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.592471 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.592314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.592471 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.592341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtp7f\" (UniqueName: \"kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.592471 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.592356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.592598 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.592462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.693880 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.693837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.693880 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.693885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtp7f\" (UniqueName: \"kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694170 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.693909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694170 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.693975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694170 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.694003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694345 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.694292    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.694580    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.694665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.694583    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.696435 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.696418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.702398 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.702376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtp7f\" (UniqueName: \"kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.827409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.827314    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:25.958986 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:25.958954    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:29:25.961328 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:29:25.961302    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa1d079_8177_432f_a308_9f768a89f888.slice/crio-7bbafc86b2e32490bcdd5bfd71ba0c633a8a028ae865a334353e14beb79d211d WatchSource:0}: Error finding container 7bbafc86b2e32490bcdd5bfd71ba0c633a8a028ae865a334353e14beb79d211d: Status 404 returned error can't find the container with id 7bbafc86b2e32490bcdd5bfd71ba0c633a8a028ae865a334353e14beb79d211d
Jul 28 10:29:26.036725 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:26.036673    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerStarted","Data":"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"}
Jul 28 10:29:26.036911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:26.036754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerStarted","Data":"7bbafc86b2e32490bcdd5bfd71ba0c633a8a028ae865a334353e14beb79d211d"}
Jul 28 10:29:56.153408 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:56.153371    2569 generic.go:356] "Generic (PLEG): container finished" podID="8aa1d079-8177-432f-a308-9f768a89f888" containerID="906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13" exitCode=0
Jul 28 10:29:56.153961 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:56.153449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerDied","Data":"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"}
Jul 28 10:29:57.159662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:57.159623    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerStarted","Data":"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"}
Jul 28 10:29:57.160090 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:57.159704    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerStarted","Data":"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"}
Jul 28 10:29:57.160090 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:57.159915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:57.160175 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:57.160088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:29:57.182971 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:29:57.182913    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" podStartSLOduration=2.631834559 podStartE2EDuration="32.182897438s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.551062879s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:29:57.18077071 +0000 UTC m=+1355.101474761" watchObservedRunningTime="2026-07-28 10:29:57.182897438 +0000 UTC m=+1355.103601486"
Jul 28 10:30:03.169854 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:30:03.169821    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:31:30.566471 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:30.566444    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:31:41.602909 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:41.602855    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 10:31:42.282804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:42.282766    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:31:42.283128 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:42.283069    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kserve-container" containerID="cri-o://1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7" gracePeriod=30
Jul 28 10:31:42.283248 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:42.283100    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kube-rbac-proxy" containerID="cri-o://85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312" gracePeriod=30
Jul 28 10:31:42.579802 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:42.579712    2569 generic.go:356] "Generic (PLEG): container finished" podID="8aa1d079-8177-432f-a308-9f768a89f888" containerID="85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312" exitCode=2
Jul 28 10:31:42.579802 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:42.579772    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerDied","Data":"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"}
Jul 28 10:31:43.165618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.165574    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 10:31:43.735753 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.735727    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:31:43.747319 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747291    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location\") pod \"8aa1d079-8177-432f-a308-9f768a89f888\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") "
Jul 28 10:31:43.747491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747333    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtp7f\" (UniqueName: \"kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f\") pod \"8aa1d079-8177-432f-a308-9f768a89f888\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") "
Jul 28 10:31:43.747491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747349    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config\") pod \"8aa1d079-8177-432f-a308-9f768a89f888\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") "
Jul 28 10:31:43.747491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert\") pod \"8aa1d079-8177-432f-a308-9f768a89f888\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") "
Jul 28 10:31:43.747491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747461    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls\") pod \"8aa1d079-8177-432f-a308-9f768a89f888\" (UID: \"8aa1d079-8177-432f-a308-9f768a89f888\") "
Jul 28 10:31:43.747748 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747701    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8aa1d079-8177-432f-a308-9f768a89f888" (UID: "8aa1d079-8177-432f-a308-9f768a89f888"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:43.747798 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.747751    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8aa1d079-8177-432f-a308-9f768a89f888" (UID: "8aa1d079-8177-432f-a308-9f768a89f888"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:43.749388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.749357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8aa1d079-8177-432f-a308-9f768a89f888" (UID: "8aa1d079-8177-432f-a308-9f768a89f888"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:43.749499 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.749471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f" (OuterVolumeSpecName: "kube-api-access-wtp7f") pod "8aa1d079-8177-432f-a308-9f768a89f888" (UID: "8aa1d079-8177-432f-a308-9f768a89f888"). InnerVolumeSpecName "kube-api-access-wtp7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:43.751966 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.751946    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8aa1d079-8177-432f-a308-9f768a89f888" (UID: "8aa1d079-8177-432f-a308-9f768a89f888"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:31:43.848719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.848625    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:31:43.848719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.848657    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8aa1d079-8177-432f-a308-9f768a89f888-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:31:43.848719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.848667    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8aa1d079-8177-432f-a308-9f768a89f888-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:31:43.848719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.848675    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtp7f\" (UniqueName: \"kubernetes.io/projected/8aa1d079-8177-432f-a308-9f768a89f888-kube-api-access-wtp7f\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:31:43.848719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:43.848705    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8aa1d079-8177-432f-a308-9f768a89f888-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:31:44.591650 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.591622    2569 generic.go:356] "Generic (PLEG): container finished" podID="8aa1d079-8177-432f-a308-9f768a89f888" containerID="1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7" exitCode=0
Jul 28 10:31:44.592086 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.591676    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"
Jul 28 10:31:44.592086 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.591705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerDied","Data":"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"}
Jul 28 10:31:44.592086 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.591766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm" event={"ID":"8aa1d079-8177-432f-a308-9f768a89f888","Type":"ContainerDied","Data":"7bbafc86b2e32490bcdd5bfd71ba0c633a8a028ae865a334353e14beb79d211d"}
Jul 28 10:31:44.592086 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.591794    2569 scope.go:117] "RemoveContainer" containerID="85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"
Jul 28 10:31:44.599715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.599696    2569 scope.go:117] "RemoveContainer" containerID="1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"
Jul 28 10:31:44.606802 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.606781    2569 scope.go:117] "RemoveContainer" containerID="906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"
Jul 28 10:31:44.611348 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.611328    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:31:44.614163 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614142    2569 scope.go:117] "RemoveContainer" containerID="85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"
Jul 28 10:31:44.614412 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:31:44.614397    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312\": container with ID starting with 85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312 not found: ID does not exist" containerID="85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"
Jul 28 10:31:44.614454 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614422    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312"} err="failed to get container status \"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312\": rpc error: code = NotFound desc = could not find container \"85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312\": container with ID starting with 85864fd061bbccf126ce0630e0a978f28f1f1951ee46b0bb73d0cd09ff06f312 not found: ID does not exist"
Jul 28 10:31:44.614454 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614441    2569 scope.go:117] "RemoveContainer" containerID="1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"
Jul 28 10:31:44.614727 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:31:44.614703    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7\": container with ID starting with 1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7 not found: ID does not exist" containerID="1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"
Jul 28 10:31:44.614835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614733    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7"} err="failed to get container status \"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7\": rpc error: code = NotFound desc = could not find container \"1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7\": container with ID starting with 1477ba1ed728b07f984fca9ed3f5a24b91e778daa3bd802ef409ecd7550594f7 not found: ID does not exist"
Jul 28 10:31:44.614835 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614753    2569 scope.go:117] "RemoveContainer" containerID="906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"
Jul 28 10:31:44.614978 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:31:44.614959    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13\": container with ID starting with 906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13 not found: ID does not exist" containerID="906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"
Jul 28 10:31:44.615022 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.614986    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13"} err="failed to get container status \"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13\": rpc error: code = NotFound desc = could not find container \"906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13\": container with ID starting with 906c767ed807292676b183657b2187a8b4812cc30776abe878eedf376d108d13 not found: ID does not exist"
Jul 28 10:31:44.617087 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:44.617066    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nbskm"]
Jul 28 10:31:46.565508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:46.565467    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa1d079-8177-432f-a308-9f768a89f888" path="/var/lib/kubelet/pods/8aa1d079-8177-432f-a308-9f768a89f888/volumes"
Jul 28 10:31:47.571716 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.571668    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.571977    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kube-rbac-proxy"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.571988    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kube-rbac-proxy"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.571998    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kserve-container"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.572003    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kserve-container"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.572019    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="storage-initializer"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.572025    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="storage-initializer"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.572071    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kserve-container"
Jul 28 10:31:47.572099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.572078    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa1d079-8177-432f-a308-9f768a89f888" containerName="kube-rbac-proxy"
Jul 28 10:31:47.576635 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.576617    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.580657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.580635    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:31:47.580845 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.580749    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:31:47.580969 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.580955    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:31:47.581019 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.580989    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 10:31:47.581716 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.581700    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:31:47.586491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.586472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 10:31:47.586634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.586610    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:31:47.597812 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.597788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:31:47.678535 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.678501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.678766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.678549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.678766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.678605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.678766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.678648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmplg\" (UniqueName: \"kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.678766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.678717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.779707 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.779656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmplg\" (UniqueName: \"kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.779891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.779757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.779891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.779804    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.779891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.779848    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.779891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.779884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.780190 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.780153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.780525 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.780502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.780608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.780502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.782245 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.782226    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.791064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.791039    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmplg\" (UniqueName: \"kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:47.887282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:47.887185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:31:48.017047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:48.017025    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:31:48.019386 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:31:48.019356    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130b498d_5e6b_4042_8d93_196c9f6c4170.slice/crio-9532e52f9d809f8b635db6f074d621a8ad3f801e953b96133b4f5876d9aa6477 WatchSource:0}: Error finding container 9532e52f9d809f8b635db6f074d621a8ad3f801e953b96133b4f5876d9aa6477: Status 404 returned error can't find the container with id 9532e52f9d809f8b635db6f074d621a8ad3f801e953b96133b4f5876d9aa6477
Jul 28 10:31:48.610446 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:48.610409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerStarted","Data":"8d5bf3a1c356dd65ab336cd4a11d1be98f1354f4b9aaa9c972187ea1e69b568f"}
Jul 28 10:31:48.610886 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:31:48.610482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerStarted","Data":"9532e52f9d809f8b635db6f074d621a8ad3f801e953b96133b4f5876d9aa6477"}
Jul 28 10:32:08.691508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:08.691471    2569 generic.go:356] "Generic (PLEG): container finished" podID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerID="8d5bf3a1c356dd65ab336cd4a11d1be98f1354f4b9aaa9c972187ea1e69b568f" exitCode=0
Jul 28 10:32:08.691987 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:08.691545    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerDied","Data":"8d5bf3a1c356dd65ab336cd4a11d1be98f1354f4b9aaa9c972187ea1e69b568f"}
Jul 28 10:32:09.699133 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:09.699092    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerStarted","Data":"9a9039eb3707be8572cdf86dcb97f15ed0d306daecad563ed7df4f45f83f302f"}
Jul 28 10:32:09.699640 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:09.699156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerStarted","Data":"cfbfadd7ed001ba775db363e86dcd2bb6116169eb84e477e514867ca6f973040"}
Jul 28 10:32:09.699640 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:09.699342    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:32:09.720143 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:09.720091    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" podStartSLOduration=2.868713762 podStartE2EDuration="22.72007274s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.851358978s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:09.719038982 +0000 UTC m=+1487.639743040" watchObservedRunningTime="2026-07-28 10:32:09.72007274 +0000 UTC m=+1487.640776791"
Jul 28 10:32:10.703929 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:10.703897    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:32:16.714122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:16.714089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:32:22.507215 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:22.507188    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:32:22.509044 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:32:22.509017    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:33:46.566417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:46.566387    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:33:53.765202 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:53.765168    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:33:53.765711 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:53.765408    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kserve-container" containerID="cri-o://cfbfadd7ed001ba775db363e86dcd2bb6116169eb84e477e514867ca6f973040" gracePeriod=30
Jul 28 10:33:53.765711 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:53.765462    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kube-rbac-proxy" containerID="cri-o://9a9039eb3707be8572cdf86dcb97f15ed0d306daecad563ed7df4f45f83f302f" gracePeriod=30
Jul 28 10:33:54.130437 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:54.130348    2569 generic.go:356] "Generic (PLEG): container finished" podID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerID="9a9039eb3707be8572cdf86dcb97f15ed0d306daecad563ed7df4f45f83f302f" exitCode=2
Jul 28 10:33:54.130437 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:54.130421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerDied","Data":"9a9039eb3707be8572cdf86dcb97f15ed0d306daecad563ed7df4f45f83f302f"}
Jul 28 10:33:55.137113 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.137086    2569 generic.go:356] "Generic (PLEG): container finished" podID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerID="cfbfadd7ed001ba775db363e86dcd2bb6116169eb84e477e514867ca6f973040" exitCode=0
Jul 28 10:33:55.137424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.137147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerDied","Data":"cfbfadd7ed001ba775db363e86dcd2bb6116169eb84e477e514867ca6f973040"}
Jul 28 10:33:55.207353 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.207325    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:33:55.359816 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.359721    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert\") pod \"130b498d-5e6b-4042-8d93-196c9f6c4170\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") "
Jul 28 10:33:55.359816 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.359789    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config\") pod \"130b498d-5e6b-4042-8d93-196c9f6c4170\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") "
Jul 28 10:33:55.359816 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.359816    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls\") pod \"130b498d-5e6b-4042-8d93-196c9f6c4170\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") "
Jul 28 10:33:55.360060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.359869    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location\") pod \"130b498d-5e6b-4042-8d93-196c9f6c4170\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") "
Jul 28 10:33:55.360060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.359905    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmplg\" (UniqueName: \"kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg\") pod \"130b498d-5e6b-4042-8d93-196c9f6c4170\" (UID: \"130b498d-5e6b-4042-8d93-196c9f6c4170\") "
Jul 28 10:33:55.360176 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.360139    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "130b498d-5e6b-4042-8d93-196c9f6c4170" (UID: "130b498d-5e6b-4042-8d93-196c9f6c4170"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:33:55.360280 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.360254    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "130b498d-5e6b-4042-8d93-196c9f6c4170" (UID: "130b498d-5e6b-4042-8d93-196c9f6c4170"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:33:55.362028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.362004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "130b498d-5e6b-4042-8d93-196c9f6c4170" (UID: "130b498d-5e6b-4042-8d93-196c9f6c4170"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:33:55.362126 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.362102    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg" (OuterVolumeSpecName: "kube-api-access-zmplg") pod "130b498d-5e6b-4042-8d93-196c9f6c4170" (UID: "130b498d-5e6b-4042-8d93-196c9f6c4170"). InnerVolumeSpecName "kube-api-access-zmplg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:33:55.364107 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.364081    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "130b498d-5e6b-4042-8d93-196c9f6c4170" (UID: "130b498d-5e6b-4042-8d93-196c9f6c4170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:33:55.461327 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.461284    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:33:55.461327 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.461319    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130b498d-5e6b-4042-8d93-196c9f6c4170-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:33:55.461327 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.461333    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/130b498d-5e6b-4042-8d93-196c9f6c4170-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:33:55.461576 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.461345    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmplg\" (UniqueName: \"kubernetes.io/projected/130b498d-5e6b-4042-8d93-196c9f6c4170-kube-api-access-zmplg\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:33:55.461576 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:55.461358    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/130b498d-5e6b-4042-8d93-196c9f6c4170-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:33:56.143340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.143299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4" event={"ID":"130b498d-5e6b-4042-8d93-196c9f6c4170","Type":"ContainerDied","Data":"9532e52f9d809f8b635db6f074d621a8ad3f801e953b96133b4f5876d9aa6477"}
Jul 28 10:33:56.143340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.143339    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"
Jul 28 10:33:56.143902 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.143381    2569 scope.go:117] "RemoveContainer" containerID="9a9039eb3707be8572cdf86dcb97f15ed0d306daecad563ed7df4f45f83f302f"
Jul 28 10:33:56.151890 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.151870    2569 scope.go:117] "RemoveContainer" containerID="cfbfadd7ed001ba775db363e86dcd2bb6116169eb84e477e514867ca6f973040"
Jul 28 10:33:56.159403 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.159384    2569 scope.go:117] "RemoveContainer" containerID="8d5bf3a1c356dd65ab336cd4a11d1be98f1354f4b9aaa9c972187ea1e69b568f"
Jul 28 10:33:56.165772 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.165745    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:33:56.172157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.172135    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g48b4"]
Jul 28 10:33:56.566743 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:56.566712    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" path="/var/lib/kubelet/pods/130b498d-5e6b-4042-8d93-196c9f6c4170/volumes"
Jul 28 10:33:58.937426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937389    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937719    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kube-rbac-proxy"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937735    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kube-rbac-proxy"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937767    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="storage-initializer"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937773    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="storage-initializer"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937782    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kserve-container"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937787    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kserve-container"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937841    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kserve-container"
Jul 28 10:33:58.937847 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.937851    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="130b498d-5e6b-4042-8d93-196c9f6c4170" containerName="kube-rbac-proxy"
Jul 28 10:33:58.942768 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.942751    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:58.945526 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945500    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:33:58.945674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945610    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 10:33:58.945674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945640    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:33:58.945674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945663    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 10:33:58.945929 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945911    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:33:58.945929 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945922    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:33:58.946098 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.945942    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:33:58.953252 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:58.953231    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:33:59.089345 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.089293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnngq\" (UniqueName: \"kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.089562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.089358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.089562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.089394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.089562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.089435    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.089562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.089463    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.190703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.190580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.190703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.190640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.190934 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.190716    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.190934 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.190749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.190934 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.190824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnngq\" (UniqueName: \"kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.191268 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.191231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.191414 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.191396    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.191462 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.191423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.193196 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.193165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.200343 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.200321    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnngq\" (UniqueName: \"kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.253370 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.253330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:33:59.380346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:33:59.380321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:33:59.382828 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:33:59.382800    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0194ff7_9a3c_4461_a700_b242acb0b379.slice/crio-6b32af4e0cd6f6dfda4c424a0d28c4bf0b4c2863a360c20c9c4060d3d390cf1b WatchSource:0}: Error finding container 6b32af4e0cd6f6dfda4c424a0d28c4bf0b4c2863a360c20c9c4060d3d390cf1b: Status 404 returned error can't find the container with id 6b32af4e0cd6f6dfda4c424a0d28c4bf0b4c2863a360c20c9c4060d3d390cf1b
Jul 28 10:34:00.160853 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:00.160816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerStarted","Data":"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"}
Jul 28 10:34:00.161297 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:00.160877    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerStarted","Data":"6b32af4e0cd6f6dfda4c424a0d28c4bf0b4c2863a360c20c9c4060d3d390cf1b"}
Jul 28 10:34:09.196118 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:09.196082    2569 generic.go:356] "Generic (PLEG): container finished" podID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerID="31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67" exitCode=0
Jul 28 10:34:09.196534 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:09.196156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerDied","Data":"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"}
Jul 28 10:34:10.211109 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:10.211068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerStarted","Data":"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"}
Jul 28 10:34:10.211496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:10.211162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerStarted","Data":"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"}
Jul 28 10:34:10.211496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:10.211365    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:10.232275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:10.232230    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podStartSLOduration=2.8815423 podStartE2EDuration="12.232214737s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.350672437s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:10.230812757 +0000 UTC m=+1608.151516807" watchObservedRunningTime="2026-07-28 10:34:10.232214737 +0000 UTC m=+1608.152918797"
Jul 28 10:34:11.216106 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:11.216078    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:11.217116 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:11.217082    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:34:12.219959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:12.219911    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:34:17.225654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:17.225578    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:17.226178 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:17.226149    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:34:27.226900 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:27.226861    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:41.388971 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:41.388933    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:34:41.389503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:41.389184    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container" containerID="cri-o://fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847" gracePeriod=30
Jul 28 10:34:41.389503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:41.389211    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kube-rbac-proxy" containerID="cri-o://06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6" gracePeriod=30
Jul 28 10:34:42.220735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:42.220662    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 10:34:42.347311 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:42.347279    2569 generic.go:356] "Generic (PLEG): container finished" podID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerID="06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6" exitCode=2
Jul 28 10:34:42.347482 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:42.347322    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerDied","Data":"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"}
Jul 28 10:34:43.133887 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.133863    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:43.246787 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.246676    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls\") pod \"d0194ff7-9a3c-4461-a700-b242acb0b379\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") "
Jul 28 10:34:43.246787 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.246740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location\") pod \"d0194ff7-9a3c-4461-a700-b242acb0b379\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") "
Jul 28 10:34:43.246787 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.246761    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config\") pod \"d0194ff7-9a3c-4461-a700-b242acb0b379\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") "
Jul 28 10:34:43.247084 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.246807    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnngq\" (UniqueName: \"kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq\") pod \"d0194ff7-9a3c-4461-a700-b242acb0b379\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") "
Jul 28 10:34:43.247084 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.246844    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert\") pod \"d0194ff7-9a3c-4461-a700-b242acb0b379\" (UID: \"d0194ff7-9a3c-4461-a700-b242acb0b379\") "
Jul 28 10:34:43.247237 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.247212    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0194ff7-9a3c-4461-a700-b242acb0b379" (UID: "d0194ff7-9a3c-4461-a700-b242acb0b379"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:34:43.247294 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.247267    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0194ff7-9a3c-4461-a700-b242acb0b379" (UID: "d0194ff7-9a3c-4461-a700-b242acb0b379"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:34:43.247884 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.247858    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0194ff7-9a3c-4461-a700-b242acb0b379" (UID: "d0194ff7-9a3c-4461-a700-b242acb0b379"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:34:43.248811 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.248789    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0194ff7-9a3c-4461-a700-b242acb0b379" (UID: "d0194ff7-9a3c-4461-a700-b242acb0b379"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:34:43.248913 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.248877    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq" (OuterVolumeSpecName: "kube-api-access-qnngq") pod "d0194ff7-9a3c-4461-a700-b242acb0b379" (UID: "d0194ff7-9a3c-4461-a700-b242acb0b379"). InnerVolumeSpecName "kube-api-access-qnngq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:34:43.347606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.347567    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0194ff7-9a3c-4461-a700-b242acb0b379-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:34:43.347606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.347598    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:34:43.347606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.347607    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnngq\" (UniqueName: \"kubernetes.io/projected/d0194ff7-9a3c-4461-a700-b242acb0b379-kube-api-access-qnngq\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:34:43.347606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.347616    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0194ff7-9a3c-4461-a700-b242acb0b379-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:34:43.347902 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.347626    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0194ff7-9a3c-4461-a700-b242acb0b379-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:34:43.352702 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.352655    2569 generic.go:356] "Generic (PLEG): container finished" podID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerID="fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847" exitCode=0
Jul 28 10:34:43.352827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.352705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerDied","Data":"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"}
Jul 28 10:34:43.352827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.352730    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"
Jul 28 10:34:43.352827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.352768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6" event={"ID":"d0194ff7-9a3c-4461-a700-b242acb0b379","Type":"ContainerDied","Data":"6b32af4e0cd6f6dfda4c424a0d28c4bf0b4c2863a360c20c9c4060d3d390cf1b"}
Jul 28 10:34:43.352827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.352795    2569 scope.go:117] "RemoveContainer" containerID="06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"
Jul 28 10:34:43.361240 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.361223    2569 scope.go:117] "RemoveContainer" containerID="fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"
Jul 28 10:34:43.372033 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.372009    2569 scope.go:117] "RemoveContainer" containerID="31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"
Jul 28 10:34:43.377988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.377961    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:34:43.380679 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.380658    2569 scope.go:117] "RemoveContainer" containerID="06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"
Jul 28 10:34:43.381024 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:34:43.380996    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6\": container with ID starting with 06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6 not found: ID does not exist" containerID="06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"
Jul 28 10:34:43.381124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381032    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6"} err="failed to get container status \"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6\": rpc error: code = NotFound desc = could not find container \"06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6\": container with ID starting with 06f0407fcaf0461ea39183cc4e5ef2c09399e1a92721d4ee4b2bf3c1090473d6 not found: ID does not exist"
Jul 28 10:34:43.381124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381051    2569 scope.go:117] "RemoveContainer" containerID="fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"
Jul 28 10:34:43.381285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381263    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2nnt6"]
Jul 28 10:34:43.381340 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:34:43.381316    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847\": container with ID starting with fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847 not found: ID does not exist" containerID="fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"
Jul 28 10:34:43.381378 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381346    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847"} err="failed to get container status \"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847\": rpc error: code = NotFound desc = could not find container \"fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847\": container with ID starting with fd2bd55b43f7a68fbd69d997b93b14d206ae4254857c85522485588a521cf847 not found: ID does not exist"
Jul 28 10:34:43.381378 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381360    2569 scope.go:117] "RemoveContainer" containerID="31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"
Jul 28 10:34:43.381589 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:34:43.381571    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67\": container with ID starting with 31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67 not found: ID does not exist" containerID="31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"
Jul 28 10:34:43.381628 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:43.381597    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67"} err="failed to get container status \"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67\": rpc error: code = NotFound desc = could not find container \"31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67\": container with ID starting with 31e81e750e96302855c6946b8f3ae415ef09c34ca4ff9bb01cb04c2888c55e67 not found: ID does not exist"
Jul 28 10:34:44.566547 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:44.566514    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" path="/var/lib/kubelet/pods/d0194ff7-9a3c-4461-a700-b242acb0b379/volumes"
Jul 28 10:34:46.566169 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566139    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566435    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="storage-initializer"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566446    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="storage-initializer"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566460    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kube-rbac-proxy"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566466    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kube-rbac-proxy"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566485    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566491    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566538    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kube-rbac-proxy"
Jul 28 10:34:46.566551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.566545    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0194ff7-9a3c-4461-a700-b242acb0b379" containerName="kserve-container"
Jul 28 10:34:46.571541 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.571524    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.574062 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574040    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 10:34:46.574206 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 10:34:46.574206 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574050    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:34:46.574206 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:34:46.574206 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574167    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:34:46.574451 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.574217    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:34:46.575014 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.575000    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:34:46.578810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.578789    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:34:46.673084 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.673047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.673084 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.673093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.673311 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.673110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.673311 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.673190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987ts\" (UniqueName: \"kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.673311 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.673236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.773899 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.773854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.773912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.773938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.773980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987ts\" (UniqueName: \"kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.774024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:34:46.774065    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 10:34:46.774192 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:34:46.774169    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls podName:a203f94a-668c-4719-bc70-23f1a5deeac6 nodeName:}" failed. No retries permitted until 2026-07-28 10:34:47.274148201 +0000 UTC m=+1645.194852235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 10:34:46.774451 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.774398    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.774616    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.774705 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.774668    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:46.784927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:46.784903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-987ts\" (UniqueName: \"kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:47.281559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:47.281517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:47.283950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:47.283915    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:47.482262 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:47.482217    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:47.611040 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:47.611008    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:34:47.613733 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:34:47.613671    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda203f94a_668c_4719_bc70_23f1a5deeac6.slice/crio-addacc0c240b7b5e5455c5a893fa45ca757306f28697352981af0920b91e7c4f WatchSource:0}: Error finding container addacc0c240b7b5e5455c5a893fa45ca757306f28697352981af0920b91e7c4f: Status 404 returned error can't find the container with id addacc0c240b7b5e5455c5a893fa45ca757306f28697352981af0920b91e7c4f
Jul 28 10:34:48.374833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:48.374797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerStarted","Data":"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"}
Jul 28 10:34:48.375020 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:48.374863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerStarted","Data":"addacc0c240b7b5e5455c5a893fa45ca757306f28697352981af0920b91e7c4f"}
Jul 28 10:34:57.412136 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:57.412104    2569 generic.go:356] "Generic (PLEG): container finished" podID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerID="e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95" exitCode=0
Jul 28 10:34:57.412533 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:57.412163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerDied","Data":"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"}
Jul 28 10:34:58.419177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:58.419140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerStarted","Data":"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"}
Jul 28 10:34:58.419575 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:58.419206    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerStarted","Data":"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"}
Jul 28 10:34:58.419575 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:58.419461    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:58.440382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:58.440326    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podStartSLOduration=3.4767868650000002 podStartE2EDuration="12.440313867s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.963527002s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:58.439319261 +0000 UTC m=+1656.360023313" watchObservedRunningTime="2026-07-28 10:34:58.440313867 +0000 UTC m=+1656.361017918"
Jul 28 10:34:59.423731 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:59.423698    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:34:59.425088 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:34:59.425062    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:35:00.427046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:00.427003    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:35:05.431010 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:05.430981    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:35:05.431747 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:05.431716    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:35:15.432884 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:15.432853    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:35:21.812694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:21.812642    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:35:21.813161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:21.812906    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container" containerID="cri-o://84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206" gracePeriod=30
Jul 28 10:35:21.813161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:21.812963    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kube-rbac-proxy" containerID="cri-o://6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c" gracePeriod=30
Jul 28 10:35:22.524826 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:22.524786    2569 generic.go:356] "Generic (PLEG): container finished" podID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerID="6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c" exitCode=2
Jul 28 10:35:22.525028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:22.524862    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerDied","Data":"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"}
Jul 28 10:35:23.258174 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.258152    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:35:23.280107 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-987ts\" (UniqueName: \"kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts\") pod \"a203f94a-668c-4719-bc70-23f1a5deeac6\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") "
Jul 28 10:35:23.280275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280119    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config\") pod \"a203f94a-668c-4719-bc70-23f1a5deeac6\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") "
Jul 28 10:35:23.280349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280287    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location\") pod \"a203f94a-668c-4719-bc70-23f1a5deeac6\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") "
Jul 28 10:35:23.280401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280356    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert\") pod \"a203f94a-668c-4719-bc70-23f1a5deeac6\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") "
Jul 28 10:35:23.280449 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280407    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") pod \"a203f94a-668c-4719-bc70-23f1a5deeac6\" (UID: \"a203f94a-668c-4719-bc70-23f1a5deeac6\") "
Jul 28 10:35:23.280520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280488    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a203f94a-668c-4719-bc70-23f1a5deeac6" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:35:23.280716 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280659    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:35:23.280853 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.280798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a203f94a-668c-4719-bc70-23f1a5deeac6" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:35:23.281703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.281658    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a203f94a-668c-4719-bc70-23f1a5deeac6" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:35:23.282496 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.282471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts" (OuterVolumeSpecName: "kube-api-access-987ts") pod "a203f94a-668c-4719-bc70-23f1a5deeac6" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6"). InnerVolumeSpecName "kube-api-access-987ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:35:23.282761 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.282740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a203f94a-668c-4719-bc70-23f1a5deeac6" (UID: "a203f94a-668c-4719-bc70-23f1a5deeac6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:35:23.381866 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.381777    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a203f94a-668c-4719-bc70-23f1a5deeac6-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:35:23.381866 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.381807    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a203f94a-668c-4719-bc70-23f1a5deeac6-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:35:23.381866 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.381816    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987ts\" (UniqueName: \"kubernetes.io/projected/a203f94a-668c-4719-bc70-23f1a5deeac6-kube-api-access-987ts\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:35:23.381866 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.381827    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a203f94a-668c-4719-bc70-23f1a5deeac6-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:35:23.530801 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.530769    2569 generic.go:356] "Generic (PLEG): container finished" podID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerID="84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206" exitCode=0
Jul 28 10:35:23.530964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.530820    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerDied","Data":"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"}
Jul 28 10:35:23.530964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.530829    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"
Jul 28 10:35:23.530964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.530867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8" event={"ID":"a203f94a-668c-4719-bc70-23f1a5deeac6","Type":"ContainerDied","Data":"addacc0c240b7b5e5455c5a893fa45ca757306f28697352981af0920b91e7c4f"}
Jul 28 10:35:23.530964 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.530895    2569 scope.go:117] "RemoveContainer" containerID="6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"
Jul 28 10:35:23.540292 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.540270    2569 scope.go:117] "RemoveContainer" containerID="84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"
Jul 28 10:35:23.547937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.547916    2569 scope.go:117] "RemoveContainer" containerID="e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"
Jul 28 10:35:23.554775 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.554758    2569 scope.go:117] "RemoveContainer" containerID="6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"
Jul 28 10:35:23.555023 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:35:23.555003    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c\": container with ID starting with 6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c not found: ID does not exist" containerID="6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"
Jul 28 10:35:23.555075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.555034    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c"} err="failed to get container status \"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c\": rpc error: code = NotFound desc = could not find container \"6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c\": container with ID starting with 6e39349f61588767fbd471cbd6fa2109e26ee3bb890364f75fd2aae5e2fc696c not found: ID does not exist"
Jul 28 10:35:23.555075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.555051    2569 scope.go:117] "RemoveContainer" containerID="84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"
Jul 28 10:35:23.555308 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:35:23.555290    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206\": container with ID starting with 84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206 not found: ID does not exist" containerID="84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"
Jul 28 10:35:23.555363 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.555313    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206"} err="failed to get container status \"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206\": rpc error: code = NotFound desc = could not find container \"84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206\": container with ID starting with 84ed5b1dbbb80c93ff65c0df4c9be77538f6a4d79586daab5b6c38a1aad3a206 not found: ID does not exist"
Jul 28 10:35:23.555363 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.555327    2569 scope.go:117] "RemoveContainer" containerID="e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"
Jul 28 10:35:23.555526 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:35:23.555506    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95\": container with ID starting with e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95 not found: ID does not exist" containerID="e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"
Jul 28 10:35:23.555580 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.555532    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95"} err="failed to get container status \"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95\": rpc error: code = NotFound desc = could not find container \"e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95\": container with ID starting with e07e646e402fc39d90f55fa3ef37c1daf18cc95c42dfec67f13341e27bebbe95 not found: ID does not exist"
Jul 28 10:35:23.560463 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.560442    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:35:23.565396 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:23.565372    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-6rmn8"]
Jul 28 10:35:24.565495 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:24.565457    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" path="/var/lib/kubelet/pods/a203f94a-668c-4719-bc70-23f1a5deeac6/volumes"
Jul 28 10:35:27.022812 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.022779    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:35:27.023248 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023231    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container"
Jul 28 10:35:27.023331 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023250    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container"
Jul 28 10:35:27.023331 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023286    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kube-rbac-proxy"
Jul 28 10:35:27.023331 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023295    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kube-rbac-proxy"
Jul 28 10:35:27.023331 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023312    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="storage-initializer"
Jul 28 10:35:27.023331 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="storage-initializer"
Jul 28 10:35:27.023589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023400    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kserve-container"
Jul 28 10:35:27.023589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.023413    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a203f94a-668c-4719-bc70-23f1a5deeac6" containerName="kube-rbac-proxy"
Jul 28 10:35:27.026432 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.026412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.031485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.031460    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 10:35:27.031821 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.031798    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:35:27.031927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.031832    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:35:27.031927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.031869    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 10:35:27.031927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.031803    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:35:27.032148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.032128    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:35:27.032270 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.032212    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:35:27.042376 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.042349    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:35:27.070236 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.070205    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:35:27.073848 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.073831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.076381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.076358    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 10:35:27.086032 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.086011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:35:27.107565 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.107721 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.107721 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.107721 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.107721 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.107878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107747    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95bv\" (UniqueName: \"kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.107878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.107878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.107878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4n9\" (UniqueName: \"kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.107878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.107872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.208854 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.208820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.208854 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.208853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4n9\" (UniqueName: \"kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.208877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.208904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.208922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209372 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209137    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209372 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:35:27.209202    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 10:35:27.209372 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:35:27.209273    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls podName:364c185d-fff4-4313-93a2-14f48ebac975 nodeName:}" failed. No retries permitted until 2026-07-28 10:35:27.709252477 +0000 UTC m=+1685.629956525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls") pod "isvc-canary-create-predictor-69fc457c9b-w4njg" (UID: "364c185d-fff4-4313-93a2-14f48ebac975") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 10:35:27.209372 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95bv\" (UniqueName: \"kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209398    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209706    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.209751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209722    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.209937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.209782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.210049 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.210029    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.211489 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.211468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.219678 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.219658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4n9\" (UniqueName: \"kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.219810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.219752    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95bv\" (UniqueName: \"kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv\") pod \"isvc-canary-create-v2-predictor-5957db6ddf-qj5c5\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.383155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.383066    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:27.515836 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.515811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:35:27.518115 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:35:27.518086    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5141e2_3c7e_42bf_a485_bf216a4d499e.slice/crio-8b71c5a67367f581563904ca680d0b41eb722113e4750c060513a5dd35622434 WatchSource:0}: Error finding container 8b71c5a67367f581563904ca680d0b41eb722113e4750c060513a5dd35622434: Status 404 returned error can't find the container with id 8b71c5a67367f581563904ca680d0b41eb722113e4750c060513a5dd35622434
Jul 28 10:35:27.548330 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.548301    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerStarted","Data":"8b71c5a67367f581563904ca680d0b41eb722113e4750c060513a5dd35622434"}
Jul 28 10:35:27.713423 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.713380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.715802 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.715775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") pod \"isvc-canary-create-predictor-69fc457c9b-w4njg\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:27.936573 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:27.936532    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:28.070584 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:28.070560    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:35:28.073560 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:35:28.073536    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364c185d_fff4_4313_93a2_14f48ebac975.slice/crio-aebec4d5713b6651a5ed2b4d2bd44c745ac5c62075b6c6c72b8189cb6660742d WatchSource:0}: Error finding container aebec4d5713b6651a5ed2b4d2bd44c745ac5c62075b6c6c72b8189cb6660742d: Status 404 returned error can't find the container with id aebec4d5713b6651a5ed2b4d2bd44c745ac5c62075b6c6c72b8189cb6660742d
Jul 28 10:35:28.554940 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:28.554905    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerStarted","Data":"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"}
Jul 28 10:35:28.556491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:28.556463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerStarted","Data":"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"}
Jul 28 10:35:28.556637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:28.556514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerStarted","Data":"aebec4d5713b6651a5ed2b4d2bd44c745ac5c62075b6c6c72b8189cb6660742d"}
Jul 28 10:35:31.570045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:31.570008    2569 generic.go:356] "Generic (PLEG): container finished" podID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerID="97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b" exitCode=0
Jul 28 10:35:31.570407 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:31.570079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerDied","Data":"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"}
Jul 28 10:35:32.575487 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.575451    2569 generic.go:356] "Generic (PLEG): container finished" podID="364c185d-fff4-4313-93a2-14f48ebac975" containerID="af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee" exitCode=0
Jul 28 10:35:32.575938 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.575523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerDied","Data":"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"}
Jul 28 10:35:32.577664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.577643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerStarted","Data":"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"}
Jul 28 10:35:32.577789 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.577710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerStarted","Data":"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"}
Jul 28 10:35:32.577993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.577976    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:32.578110 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.578020    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:32.579244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.579220    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:35:32.619974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:32.619915    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podStartSLOduration=1.707681703 podStartE2EDuration="5.619900861s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912219158s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:32.617994347 +0000 UTC m=+1690.538698401" watchObservedRunningTime="2026-07-28 10:35:32.619900861 +0000 UTC m=+1690.540604913"
Jul 28 10:35:33.584595 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:33.584561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerStarted","Data":"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"}
Jul 28 10:35:33.584994 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:33.584624    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerStarted","Data":"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"}
Jul 28 10:35:33.584994 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:33.584940    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:35:33.604856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:33.604809    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podStartSLOduration=2.999660989 podStartE2EDuration="6.604792753s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.605131764s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:33.603807886 +0000 UTC m=+1691.524511939" watchObservedRunningTime="2026-07-28 10:35:33.604792753 +0000 UTC m=+1691.525496804"
Jul 28 10:35:38.585462 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.585421    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:38.586006 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.585911    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:38.586672 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.586639    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:35:38.589586 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.589567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:35:38.590045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.590026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:35:38.590045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.590035    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:35:38.608657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:38.608626    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:35:39.612895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:39.612852    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:35:48.590185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:48.590103    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:35:49.613470 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:49.613426    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:35:58.590401 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:58.590360    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:35:59.613642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:35:59.613596    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:08.590739 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:08.590657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:36:09.613124 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:09.613075    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:18.590918 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:18.590878    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:36:19.613238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:19.613197    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:28.590384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:28.590337    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:36:29.612879 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:29.612838    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:38.590067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:38.590020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:36:39.613097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:39.613053    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:47.561743 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:47.561712    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:36:48.590728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:48.590699    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:36:57.143741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.143708    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:36:57.144260 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.143939    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" containerID="cri-o://6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888" gracePeriod=30
Jul 28 10:36:57.144260 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.144005    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kube-rbac-proxy" containerID="cri-o://baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e" gracePeriod=30
Jul 28 10:36:57.150028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.150003    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:36:57.150303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.150260    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" containerID="cri-o://47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565" gracePeriod=30
Jul 28 10:36:57.150355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.150322    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kube-rbac-proxy" containerID="cri-o://a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b" gracePeriod=30
Jul 28 10:36:57.561790 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.561747    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:36:57.949741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.949674    2569 generic.go:356] "Generic (PLEG): container finished" podID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerID="a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b" exitCode=2
Jul 28 10:36:57.949741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.949723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerDied","Data":"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"}
Jul 28 10:36:57.951737 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.951716    2569 generic.go:356] "Generic (PLEG): container finished" podID="364c185d-fff4-4313-93a2-14f48ebac975" containerID="baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e" exitCode=2
Jul 28 10:36:57.951860 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:57.951749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerDied","Data":"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"}
Jul 28 10:36:58.585671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:58.585636    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 10:36:58.589990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:58.589965    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:36:58.590342 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:36:58.590318    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 10:37:02.218609 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.218513    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:37:02.221651 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.221629    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:37:02.223760 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.223741    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config\") pod \"364c185d-fff4-4313-93a2-14f48ebac975\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") "
Jul 28 10:37:02.223901 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.223883    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl4n9\" (UniqueName: \"kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9\") pod \"364c185d-fff4-4313-93a2-14f48ebac975\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") "
Jul 28 10:37:02.223970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.223923    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location\") pod \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") "
Jul 28 10:37:02.224034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.223976    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert\") pod \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") "
Jul 28 10:37:02.224034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224010    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") pod \"364c185d-fff4-4313-93a2-14f48ebac975\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") "
Jul 28 10:37:02.224034 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "364c185d-fff4-4313-93a2-14f48ebac975" (UID: "364c185d-fff4-4313-93a2-14f48ebac975"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:37:02.224185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224045    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config\") pod \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") "
Jul 28 10:37:02.224185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224086    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location\") pod \"364c185d-fff4-4313-93a2-14f48ebac975\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") "
Jul 28 10:37:02.224185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls\") pod \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") "
Jul 28 10:37:02.224185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k95bv\" (UniqueName: \"kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv\") pod \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\" (UID: \"ee5141e2-3c7e-42bf-a485-bf216a4d499e\") "
Jul 28 10:37:02.224382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224301    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee5141e2-3c7e-42bf-a485-bf216a4d499e" (UID: "ee5141e2-3c7e-42bf-a485-bf216a4d499e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:02.224382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224339    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee5141e2-3c7e-42bf-a485-bf216a4d499e" (UID: "ee5141e2-3c7e-42bf-a485-bf216a4d499e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:37:02.224532 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee5141e2-3c7e-42bf-a485-bf216a4d499e" (UID: "ee5141e2-3c7e-42bf-a485-bf216a4d499e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:37:02.224532 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "364c185d-fff4-4313-93a2-14f48ebac975" (UID: "364c185d-fff4-4313-93a2-14f48ebac975"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:37:02.224532 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224524    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.224666 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224542    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.224666 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.224559    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.226317 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.226297    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee5141e2-3c7e-42bf-a485-bf216a4d499e" (UID: "ee5141e2-3c7e-42bf-a485-bf216a4d499e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:37:02.226764 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.226744    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv" (OuterVolumeSpecName: "kube-api-access-k95bv") pod "ee5141e2-3c7e-42bf-a485-bf216a4d499e" (UID: "ee5141e2-3c7e-42bf-a485-bf216a4d499e"). InnerVolumeSpecName "kube-api-access-k95bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:37:02.226764 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.226754    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9" (OuterVolumeSpecName: "kube-api-access-vl4n9") pod "364c185d-fff4-4313-93a2-14f48ebac975" (UID: "364c185d-fff4-4313-93a2-14f48ebac975"). InnerVolumeSpecName "kube-api-access-vl4n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:37:02.226870 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.226777    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "364c185d-fff4-4313-93a2-14f48ebac975" (UID: "364c185d-fff4-4313-93a2-14f48ebac975"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:37:02.317225 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317146    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:37:02.317465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317453    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kube-rbac-proxy"
Jul 28 10:37:02.317504 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317467    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kube-rbac-proxy"
Jul 28 10:37:02.317504 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317482    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container"
Jul 28 10:37:02.317504 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317488    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container"
Jul 28 10:37:02.317504 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317498    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="storage-initializer"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317504    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="storage-initializer"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317514    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kube-rbac-proxy"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317521    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kube-rbac-proxy"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317531    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317537    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317547    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="storage-initializer"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317552    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="storage-initializer"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317600    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kserve-container"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317614    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kube-rbac-proxy"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317620    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerName="kserve-container"
Jul 28 10:37:02.317634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.317627    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="364c185d-fff4-4313-93a2-14f48ebac975" containerName="kube-rbac-proxy"
Jul 28 10:37:02.321983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.321968    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.324347 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324322    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 10:37:02.324482 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324468    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 10:37:02.324831 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert\") pod \"364c185d-fff4-4313-93a2-14f48ebac975\" (UID: \"364c185d-fff4-4313-93a2-14f48ebac975\") "
Jul 28 10:37:02.324934 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324920    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhd5w\" (UniqueName: \"kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.324999 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.325055 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.324999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.325120 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.325120 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.325222 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325160    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95bv\" (UniqueName: \"kubernetes.io/projected/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-api-access-k95bv\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.325222 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325192    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "364c185d-fff4-4313-93a2-14f48ebac975" (UID: "364c185d-fff4-4313-93a2-14f48ebac975"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:37:02.325318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325238    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl4n9\" (UniqueName: \"kubernetes.io/projected/364c185d-fff4-4313-93a2-14f48ebac975-kube-api-access-vl4n9\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.325318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325262    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/364c185d-fff4-4313-93a2-14f48ebac975-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.325318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325278    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee5141e2-3c7e-42bf-a485-bf216a4d499e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.325318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325291    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/364c185d-fff4-4313-93a2-14f48ebac975-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.325318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.325305    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee5141e2-3c7e-42bf-a485-bf216a4d499e-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.327846 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.327825    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:37:02.331982 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.331960    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.333018 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.332997    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:37:02.334488 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.334474    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 10:37:02.342918 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.342896    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:37:02.426102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426065    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426250    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhd5w\" (UniqueName: \"kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426269    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.426778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sslgb\" (UniqueName: \"kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.426778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.426778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.426778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426537    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/364c185d-fff4-4313-93a2-14f48ebac975-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:37:02.426778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.426956 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.426909    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.428637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.428619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.435512 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.435488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhd5w\" (UniqueName: \"kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w\") pod \"isvc-canary-promote-predictor-698bbb5b56-5qkrf\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.526948 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.526914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.526962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sslgb\" (UniqueName: \"kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.526999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527101 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.527027    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527101 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:02.527074    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 10:37:02.527309 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:02.527140    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls podName:3a20be4d-5f84-420b-8630-e4e387e9f23f nodeName:}" failed. No retries permitted until 2026-07-28 10:37:03.027118601 +0000 UTC m=+1780.947822639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls") pod "isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 10:37:02.527309 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.527158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527556 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.527534    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527729 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.527712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.527774 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.527740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.536095 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.536056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sslgb\" (UniqueName: \"kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:02.633647 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.633609    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:02.777851 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.777827    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:37:02.780486 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:37:02.780460    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72ab86a_1611_47ce_8c20_3b32fc642b4d.slice/crio-529fdb6b6b2ad8bd64f3ed85dee2a78b599ff2bec8ab259d091ef65803dc557e WatchSource:0}: Error finding container 529fdb6b6b2ad8bd64f3ed85dee2a78b599ff2bec8ab259d091ef65803dc557e: Status 404 returned error can't find the container with id 529fdb6b6b2ad8bd64f3ed85dee2a78b599ff2bec8ab259d091ef65803dc557e
Jul 28 10:37:02.973145 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.973108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerStarted","Data":"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"}
Jul 28 10:37:02.973344 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.973176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerStarted","Data":"529fdb6b6b2ad8bd64f3ed85dee2a78b599ff2bec8ab259d091ef65803dc557e"}
Jul 28 10:37:02.975121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.975097    2569 generic.go:356] "Generic (PLEG): container finished" podID="364c185d-fff4-4313-93a2-14f48ebac975" containerID="6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888" exitCode=0
Jul 28 10:37:02.975243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.975184    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"
Jul 28 10:37:02.975310 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.975187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerDied","Data":"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"}
Jul 28 10:37:02.975370 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.975328    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg" event={"ID":"364c185d-fff4-4313-93a2-14f48ebac975","Type":"ContainerDied","Data":"aebec4d5713b6651a5ed2b4d2bd44c745ac5c62075b6c6c72b8189cb6660742d"}
Jul 28 10:37:02.975370 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.975360    2569 scope.go:117] "RemoveContainer" containerID="baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"
Jul 28 10:37:02.977384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.977361    2569 generic.go:356] "Generic (PLEG): container finished" podID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" containerID="47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565" exitCode=0
Jul 28 10:37:02.977487 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.977439    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerDied","Data":"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"}
Jul 28 10:37:02.977582 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.977485    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5" event={"ID":"ee5141e2-3c7e-42bf-a485-bf216a4d499e","Type":"ContainerDied","Data":"8b71c5a67367f581563904ca680d0b41eb722113e4750c060513a5dd35622434"}
Jul 28 10:37:02.977582 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.977519    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"
Jul 28 10:37:02.985843 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.985825    2569 scope.go:117] "RemoveContainer" containerID="6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"
Jul 28 10:37:02.994832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:02.994814    2569 scope.go:117] "RemoveContainer" containerID="af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"
Jul 28 10:37:03.005137 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.005113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:37:03.014436 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.014405    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5957db6ddf-qj5c5"]
Jul 28 10:37:03.022755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.022729    2569 scope.go:117] "RemoveContainer" containerID="baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"
Jul 28 10:37:03.023124 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.023089    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e\": container with ID starting with baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e not found: ID does not exist" containerID="baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"
Jul 28 10:37:03.023213 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023136    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e"} err="failed to get container status \"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e\": rpc error: code = NotFound desc = could not find container \"baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e\": container with ID starting with baa202157f4b19db2525741bca51c2f685cbdd23d060631c6fe3b71a7f219e4e not found: ID does not exist"
Jul 28 10:37:03.023213 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023162    2569 scope.go:117] "RemoveContainer" containerID="6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"
Jul 28 10:37:03.023450 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.023429    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888\": container with ID starting with 6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888 not found: ID does not exist" containerID="6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"
Jul 28 10:37:03.023509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023457    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888"} err="failed to get container status \"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888\": rpc error: code = NotFound desc = could not find container \"6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888\": container with ID starting with 6c63e1ee28046ef5c8e81f15650172d5d0328adaf574254db00e2838ec86c888 not found: ID does not exist"
Jul 28 10:37:03.023509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023474    2569 scope.go:117] "RemoveContainer" containerID="af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"
Jul 28 10:37:03.023830 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.023807    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee\": container with ID starting with af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee not found: ID does not exist" containerID="af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"
Jul 28 10:37:03.023903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023839    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee"} err="failed to get container status \"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee\": rpc error: code = NotFound desc = could not find container \"af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee\": container with ID starting with af50b6b5edb02667d912263edf72b0f245f3f2af3386fac5935a8bd612190eee not found: ID does not exist"
Jul 28 10:37:03.023903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.023859    2569 scope.go:117] "RemoveContainer" containerID="a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"
Jul 28 10:37:03.024165 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.024144    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:37:03.028474 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.028454    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-69fc457c9b-w4njg"]
Jul 28 10:37:03.030537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.030516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:03.032362 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.032346    2569 scope.go:117] "RemoveContainer" containerID="47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"
Jul 28 10:37:03.033198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.033176    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:03.043257 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.043233    2569 scope.go:117] "RemoveContainer" containerID="97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"
Jul 28 10:37:03.051027 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051008    2569 scope.go:117] "RemoveContainer" containerID="a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"
Jul 28 10:37:03.051312 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.051294    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b\": container with ID starting with a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b not found: ID does not exist" containerID="a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"
Jul 28 10:37:03.051391 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051323    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b"} err="failed to get container status \"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b\": rpc error: code = NotFound desc = could not find container \"a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b\": container with ID starting with a6af68d1e533ffa10f122c8ce8a331016b0aba715d3ed241a824f7c593239d3b not found: ID does not exist"
Jul 28 10:37:03.051391 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051343    2569 scope.go:117] "RemoveContainer" containerID="47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"
Jul 28 10:37:03.051605 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.051588    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565\": container with ID starting with 47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565 not found: ID does not exist" containerID="47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"
Jul 28 10:37:03.051645 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051611    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565"} err="failed to get container status \"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565\": rpc error: code = NotFound desc = could not find container \"47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565\": container with ID starting with 47a793fe8d9fb2e737c240d6ff9bdcccb58dc01f8d046449aafdd14412bce565 not found: ID does not exist"
Jul 28 10:37:03.051645 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051631    2569 scope.go:117] "RemoveContainer" containerID="97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"
Jul 28 10:37:03.051922 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:37:03.051905    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b\": container with ID starting with 97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b not found: ID does not exist" containerID="97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"
Jul 28 10:37:03.051977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.051925    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b"} err="failed to get container status \"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b\": rpc error: code = NotFound desc = could not find container \"97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b\": container with ID starting with 97d6a9084c60bb18d724ec2683b48c28d7751ff5004223b40b9afc22c0c9849b not found: ID does not exist"
Jul 28 10:37:03.242911 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.242877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:03.385855 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.385819    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:37:03.388974 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:37:03.388942    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a20be4d_5f84_420b_8630_e4e387e9f23f.slice/crio-b36460e2f1af4f38c82b4deca2a6e01134eba0be2921209abc31ff40f785bad9 WatchSource:0}: Error finding container b36460e2f1af4f38c82b4deca2a6e01134eba0be2921209abc31ff40f785bad9: Status 404 returned error can't find the container with id b36460e2f1af4f38c82b4deca2a6e01134eba0be2921209abc31ff40f785bad9
Jul 28 10:37:03.984196 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.984159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerStarted","Data":"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"}
Jul 28 10:37:03.984429 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:03.984230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerStarted","Data":"b36460e2f1af4f38c82b4deca2a6e01134eba0be2921209abc31ff40f785bad9"}
Jul 28 10:37:04.566810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:04.566777    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364c185d-fff4-4313-93a2-14f48ebac975" path="/var/lib/kubelet/pods/364c185d-fff4-4313-93a2-14f48ebac975/volumes"
Jul 28 10:37:04.567285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:04.567272    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5141e2-3c7e-42bf-a485-bf216a4d499e" path="/var/lib/kubelet/pods/ee5141e2-3c7e-42bf-a485-bf216a4d499e/volumes"
Jul 28 10:37:06.998483 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:06.998456    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerID="20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89" exitCode=0
Jul 28 10:37:06.998885 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:06.998527    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerDied","Data":"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"}
Jul 28 10:37:07.000172 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:07.000143    2569 generic.go:356] "Generic (PLEG): container finished" podID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerID="cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85" exitCode=0
Jul 28 10:37:07.000271 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:07.000221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerDied","Data":"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"}
Jul 28 10:37:08.007247 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.007208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerStarted","Data":"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"}
Jul 28 10:37:08.007665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.007280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerStarted","Data":"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"}
Jul 28 10:37:08.007665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.007526    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:08.007665 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.007584    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:08.008876 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.008845    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:08.009587 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.009567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerStarted","Data":"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"}
Jul 28 10:37:08.009674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.009609    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerStarted","Data":"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"}
Jul 28 10:37:08.009879 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.009861    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:08.009950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.009907    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:08.010826 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.010805    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:08.027474 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.027430    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podStartSLOduration=2.508071794 podStartE2EDuration="6.027415987s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.519344193s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:08.025933342 +0000 UTC m=+1785.946637396" watchObservedRunningTime="2026-07-28 10:37:08.027415987 +0000 UTC m=+1785.948120039"
Jul 28 10:37:08.046140 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:08.046097    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podStartSLOduration=2.3676596 podStartE2EDuration="6.046082009s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.678422409s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:08.044338711 +0000 UTC m=+1785.965042764" watchObservedRunningTime="2026-07-28 10:37:08.046082009 +0000 UTC m=+1785.966786061"
Jul 28 10:37:09.013327 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:09.013288    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:09.013744 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:09.013408    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:14.019007 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:14.018971    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:37:14.019419 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:14.019161    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:37:14.019419 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:14.019376    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:14.019990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:14.019964    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:22.532994 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:22.532966    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:37:22.535710 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:22.535673    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:37:24.019930 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:24.019895    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:24.020328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:24.019895    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:34.019642 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:34.019602    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:34.020123 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:34.019881    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:44.020126 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:44.020087    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:37:44.020508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:44.020087    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:54.020169 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:54.020134    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:37:54.020648 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:37:54.020135    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:38:04.019677 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:04.019641    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:38:04.020071 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:04.019872    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:38:14.019767 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:14.019727    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:38:14.020192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:14.019861    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:38:22.567327 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:22.567298    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:38:24.021421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:24.021392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:38:32.419140 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:32.419104    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:38:32.419631 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:32.419371    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" containerID="cri-o://9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7" gracePeriod=30
Jul 28 10:38:32.419631 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:32.419420    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kube-rbac-proxy" containerID="cri-o://5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b" gracePeriod=30
Jul 28 10:38:33.358366 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:33.358329    2569 generic.go:356] "Generic (PLEG): container finished" podID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerID="5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b" exitCode=2
Jul 28 10:38:33.358544 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:33.358405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerDied","Data":"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"}
Jul 28 10:38:34.014568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:34.014517    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 10:38:34.019944 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:34.019916    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:38:37.074338 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.074310    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:38:37.145869 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.145838    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhd5w\" (UniqueName: \"kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w\") pod \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") "
Jul 28 10:38:37.146025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.145885    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert\") pod \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") "
Jul 28 10:38:37.146025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.145928    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location\") pod \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") "
Jul 28 10:38:37.146025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.145951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config\") pod \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") "
Jul 28 10:38:37.146025 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.145990    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls\") pod \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\" (UID: \"d72ab86a-1611-47ce-8c20-3b32fc642b4d\") "
Jul 28 10:38:37.146321 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.146291    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d72ab86a-1611-47ce-8c20-3b32fc642b4d" (UID: "d72ab86a-1611-47ce-8c20-3b32fc642b4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:38:37.146417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.146290    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d72ab86a-1611-47ce-8c20-3b32fc642b4d" (UID: "d72ab86a-1611-47ce-8c20-3b32fc642b4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:37.146417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.146344    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d72ab86a-1611-47ce-8c20-3b32fc642b4d" (UID: "d72ab86a-1611-47ce-8c20-3b32fc642b4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:38:37.148011 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.147986    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w" (OuterVolumeSpecName: "kube-api-access-xhd5w") pod "d72ab86a-1611-47ce-8c20-3b32fc642b4d" (UID: "d72ab86a-1611-47ce-8c20-3b32fc642b4d"). InnerVolumeSpecName "kube-api-access-xhd5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:37.148111 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.148065    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d72ab86a-1611-47ce-8c20-3b32fc642b4d" (UID: "d72ab86a-1611-47ce-8c20-3b32fc642b4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:38:37.247102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.247070    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d72ab86a-1611-47ce-8c20-3b32fc642b4d-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:38:37.247102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.247094    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhd5w\" (UniqueName: \"kubernetes.io/projected/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-api-access-xhd5w\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:38:37.247102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.247106    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:38:37.247316 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.247114    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:38:37.247316 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.247123    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d72ab86a-1611-47ce-8c20-3b32fc642b4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:38:37.378326 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.378294    2569 generic.go:356] "Generic (PLEG): container finished" podID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerID="9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7" exitCode=0
Jul 28 10:38:37.378503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.378351    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"
Jul 28 10:38:37.378503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.378365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerDied","Data":"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"}
Jul 28 10:38:37.378503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.378421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf" event={"ID":"d72ab86a-1611-47ce-8c20-3b32fc642b4d","Type":"ContainerDied","Data":"529fdb6b6b2ad8bd64f3ed85dee2a78b599ff2bec8ab259d091ef65803dc557e"}
Jul 28 10:38:37.378503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.378460    2569 scope.go:117] "RemoveContainer" containerID="5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"
Jul 28 10:38:37.387535 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.387514    2569 scope.go:117] "RemoveContainer" containerID="9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"
Jul 28 10:38:37.394695 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.394668    2569 scope.go:117] "RemoveContainer" containerID="cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"
Jul 28 10:38:37.401193 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.401171    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:38:37.401817 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.401802    2569 scope.go:117] "RemoveContainer" containerID="5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"
Jul 28 10:38:37.402086 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:38:37.402045    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b\": container with ID starting with 5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b not found: ID does not exist" containerID="5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"
Jul 28 10:38:37.402086 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.402075    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b"} err="failed to get container status \"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b\": rpc error: code = NotFound desc = could not find container \"5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b\": container with ID starting with 5e6ffe93248418a0baea6989fd270e57650e5cff6414f7416945ae639424a05b not found: ID does not exist"
Jul 28 10:38:37.402169 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.402092    2569 scope.go:117] "RemoveContainer" containerID="9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"
Jul 28 10:38:37.402304 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:38:37.402289    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7\": container with ID starting with 9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7 not found: ID does not exist" containerID="9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"
Jul 28 10:38:37.402340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.402310    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7"} err="failed to get container status \"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7\": rpc error: code = NotFound desc = could not find container \"9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7\": container with ID starting with 9bbfbabd12432dafb6c56683267648fc25a6208a68fb29efc326d601a6b4b7a7 not found: ID does not exist"
Jul 28 10:38:37.402340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.402325    2569 scope.go:117] "RemoveContainer" containerID="cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"
Jul 28 10:38:37.402523 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:38:37.402509    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85\": container with ID starting with cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85 not found: ID does not exist" containerID="cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"
Jul 28 10:38:37.402558 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.402528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85"} err="failed to get container status \"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85\": rpc error: code = NotFound desc = could not find container \"cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85\": container with ID starting with cba3f97917186f684efd73c42e7ceb80a4b0365f5b73c5ac961b367a92940b85 not found: ID does not exist"
Jul 28 10:38:37.405044 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:37.405026    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-698bbb5b56-5qkrf"]
Jul 28 10:38:38.566416 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:38:38.566384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" path="/var/lib/kubelet/pods/d72ab86a-1611-47ce-8c20-3b32fc642b4d/volumes"
Jul 28 10:40:12.669081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:12.669043    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:40:12.669600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:12.669277    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container" containerID="cri-o://a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91" gracePeriod=30
Jul 28 10:40:12.669600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:12.669307    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kube-rbac-proxy" containerID="cri-o://c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764" gracePeriod=30
Jul 28 10:40:13.763289 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:13.763255    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerID="c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764" exitCode=2
Jul 28 10:40:13.763666 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:13.763327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerDied","Data":"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"}
Jul 28 10:40:14.014569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:14.014477    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 10:40:17.411047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.411023    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:40:17.560993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.560954    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location\") pod \"3a20be4d-5f84-420b-8630-e4e387e9f23f\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") "
Jul 28 10:40:17.560993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.560995    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config\") pod \"3a20be4d-5f84-420b-8630-e4e387e9f23f\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") "
Jul 28 10:40:17.561262 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sslgb\" (UniqueName: \"kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb\") pod \"3a20be4d-5f84-420b-8630-e4e387e9f23f\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") "
Jul 28 10:40:17.561262 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561094    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert\") pod \"3a20be4d-5f84-420b-8630-e4e387e9f23f\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") "
Jul 28 10:40:17.561372 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561281    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") pod \"3a20be4d-5f84-420b-8630-e4e387e9f23f\" (UID: \"3a20be4d-5f84-420b-8630-e4e387e9f23f\") "
Jul 28 10:40:17.561430 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561410    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a20be4d-5f84-420b-8630-e4e387e9f23f" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:40:17.561494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561446    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a20be4d-5f84-420b-8630-e4e387e9f23f" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:17.561543 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561488    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a20be4d-5f84-420b-8630-e4e387e9f23f" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:17.561589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561559    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a20be4d-5f84-420b-8630-e4e387e9f23f-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:40:17.561589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561576    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:40:17.561655 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.561589    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a20be4d-5f84-420b-8630-e4e387e9f23f-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:40:17.563213 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.563188    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb" (OuterVolumeSpecName: "kube-api-access-sslgb") pod "3a20be4d-5f84-420b-8630-e4e387e9f23f" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f"). InnerVolumeSpecName "kube-api-access-sslgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:17.563289 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.563199    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a20be4d-5f84-420b-8630-e4e387e9f23f" (UID: "3a20be4d-5f84-420b-8630-e4e387e9f23f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:17.662076 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.662041    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a20be4d-5f84-420b-8630-e4e387e9f23f-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:40:17.662076 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.662077    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sslgb\" (UniqueName: \"kubernetes.io/projected/3a20be4d-5f84-420b-8630-e4e387e9f23f-kube-api-access-sslgb\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:40:17.785728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.785677    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerID="a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91" exitCode=0
Jul 28 10:40:17.785728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.785717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerDied","Data":"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"}
Jul 28 10:40:17.785935 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.785771    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"
Jul 28 10:40:17.785935 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.785790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk" event={"ID":"3a20be4d-5f84-420b-8630-e4e387e9f23f","Type":"ContainerDied","Data":"b36460e2f1af4f38c82b4deca2a6e01134eba0be2921209abc31ff40f785bad9"}
Jul 28 10:40:17.785935 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.785827    2569 scope.go:117] "RemoveContainer" containerID="c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"
Jul 28 10:40:17.794523 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.794504    2569 scope.go:117] "RemoveContainer" containerID="a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"
Jul 28 10:40:17.803053 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.803031    2569 scope.go:117] "RemoveContainer" containerID="20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"
Jul 28 10:40:17.810987 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.810963    2569 scope.go:117] "RemoveContainer" containerID="c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"
Jul 28 10:40:17.811284 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:40:17.811265    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764\": container with ID starting with c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764 not found: ID does not exist" containerID="c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"
Jul 28 10:40:17.811348 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.811293    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764"} err="failed to get container status \"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764\": rpc error: code = NotFound desc = could not find container \"c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764\": container with ID starting with c53ab88e6b48f482a6d50987ac9e72194950f083736a590f514ce9b944835764 not found: ID does not exist"
Jul 28 10:40:17.811348 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.811325    2569 scope.go:117] "RemoveContainer" containerID="a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"
Jul 28 10:40:17.811594 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:40:17.811573    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91\": container with ID starting with a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91 not found: ID does not exist" containerID="a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"
Jul 28 10:40:17.811664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.811601    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91"} err="failed to get container status \"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91\": rpc error: code = NotFound desc = could not find container \"a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91\": container with ID starting with a91a47332b3c76cb8d29cf20fca8116f00f031775ab6b820bcfe9161ffcdbd91 not found: ID does not exist"
Jul 28 10:40:17.811664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.811617    2569 scope.go:117] "RemoveContainer" containerID="20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"
Jul 28 10:40:17.811937 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:40:17.811919    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89\": container with ID starting with 20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89 not found: ID does not exist" containerID="20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"
Jul 28 10:40:17.811998 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.811956    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89"} err="failed to get container status \"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89\": rpc error: code = NotFound desc = could not find container \"20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89\": container with ID starting with 20962b5eafe2659b604ec02f3f84c7ba2e6cb8aff631bd5981f58461fd6cfa89 not found: ID does not exist"
Jul 28 10:40:17.812804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.812762    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:40:17.814784 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.814761    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-ts7fk"]
Jul 28 10:40:17.871507 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871473    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:40:17.871780 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871768    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="storage-initializer"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871782    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="storage-initializer"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871791    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871797    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871810    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871816    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871827    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kube-rbac-proxy"
Jul 28 10:40:17.871832 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871833    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kube-rbac-proxy"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871841    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="storage-initializer"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871847    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="storage-initializer"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871852    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kube-rbac-proxy"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871857    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kube-rbac-proxy"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871901    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kube-rbac-proxy"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871909    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kube-rbac-proxy"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871915    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" containerName="kserve-container"
Jul 28 10:40:17.872043 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.871922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72ab86a-1611-47ce-8c20-3b32fc642b4d" containerName="kserve-container"
Jul 28 10:40:17.876599 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.876578    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:40:17.876740 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.876712    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.879795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879772    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:40:17.879922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879773    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 10:40:17.879922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879814    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:40:17.879922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879819    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 10:40:17.879922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879778    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:40:17.880100 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.879963    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:17.880290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.880276    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:40:17.880346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.880318    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:40:17.882781 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.882761    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 10:40:17.887105 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.887085    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:40:17.889782 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.889760    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:40:17.964975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.964881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjtd\" (UniqueName: \"kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.964975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.964925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts2z\" (UniqueName: \"kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:17.964975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.964949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:17.965256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965231    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:17.965457 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:17.965268    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066708 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066648    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066708 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjtd\" (UniqueName: \"kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts2z\" (UniqueName: \"kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066866    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.066981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.066944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.067399 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:40:18.067010    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 10:40:18.067399 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:40:18.067097    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls podName:93c2a29e-6569-4bb7-809a-f68046e2ea64 nodeName:}" failed. No retries permitted until 2026-07-28 10:40:18.567075519 +0000 UTC m=+1976.487779569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 10:40:18.067399 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067201    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.067399 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.067633 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.067728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.067788 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.067857 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.067836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.069559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.069542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.076240 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.076219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjtd\" (UniqueName: \"kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-ndnk9\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.076412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.076395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts2z\" (UniqueName: \"kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.188275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.188241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:18.318238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.318216    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:40:18.320243 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:40:18.320208    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c3c537_4235_48dc_b273_be0ea6cde544.slice/crio-ac93f361b073ceccabf3346cb3b750a420ec5c98ea5bd64a0f21e4254e780f45 WatchSource:0}: Error finding container ac93f361b073ceccabf3346cb3b750a420ec5c98ea5bd64a0f21e4254e780f45: Status 404 returned error can't find the container with id ac93f361b073ceccabf3346cb3b750a420ec5c98ea5bd64a0f21e4254e780f45
Jul 28 10:40:18.567315 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.567228    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a20be4d-5f84-420b-8630-e4e387e9f23f" path="/var/lib/kubelet/pods/3a20be4d-5f84-420b-8630-e4e387e9f23f/volumes"
Jul 28 10:40:18.577234 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.572144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.577234 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.575403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.794568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.794530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerStarted","Data":"b4041149c323d7809c86e00b84d588e9c8bd9e00a964e0e4aabb3237bca1ebf4"}
Jul 28 10:40:18.794756 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.794573    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:18.794756 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.794648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerStarted","Data":"ac93f361b073ceccabf3346cb3b750a420ec5c98ea5bd64a0f21e4254e780f45"}
Jul 28 10:40:18.932340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:18.932314    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:40:18.934782 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:40:18.934754    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c2a29e_6569_4bb7_809a_f68046e2ea64.slice/crio-72d0f66de932a9acc0e09679e498d65e0c06a0b15cff23de95bd6c85f48e95ee WatchSource:0}: Error finding container 72d0f66de932a9acc0e09679e498d65e0c06a0b15cff23de95bd6c85f48e95ee: Status 404 returned error can't find the container with id 72d0f66de932a9acc0e09679e498d65e0c06a0b15cff23de95bd6c85f48e95ee
Jul 28 10:40:19.801638 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:19.801601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerStarted","Data":"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"}
Jul 28 10:40:19.802038 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:19.801669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerStarted","Data":"72d0f66de932a9acc0e09679e498d65e0c06a0b15cff23de95bd6c85f48e95ee"}
Jul 28 10:40:22.815647 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:22.815609    2569 generic.go:356] "Generic (PLEG): container finished" podID="69c3c537-4235-48dc-b273-be0ea6cde544" containerID="b4041149c323d7809c86e00b84d588e9c8bd9e00a964e0e4aabb3237bca1ebf4" exitCode=0
Jul 28 10:40:22.816071 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:22.815675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerDied","Data":"b4041149c323d7809c86e00b84d588e9c8bd9e00a964e0e4aabb3237bca1ebf4"}
Jul 28 10:40:23.821010 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.820973    2569 generic.go:356] "Generic (PLEG): container finished" podID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerID="42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150" exitCode=0
Jul 28 10:40:23.821394 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.821050    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerDied","Data":"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"}
Jul 28 10:40:23.823113 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.823090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerStarted","Data":"b4d3e592591e2d4447babd0d6a959ddbd148ce11768f9d8b3cb596158625dc74"}
Jul 28 10:40:23.823219 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.823146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerStarted","Data":"07f1863e19a7db1fa4c3f7359f03efd6fe17d20441d82a9504c449c9cf38948e"}
Jul 28 10:40:23.823368 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.823349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:23.862893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:23.862847    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podStartSLOduration=3.200005179 podStartE2EDuration="6.862831264s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662826085s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:23.861637691 +0000 UTC m=+1981.782341745" watchObservedRunningTime="2026-07-28 10:40:23.862831264 +0000 UTC m=+1981.783535316"
Jul 28 10:40:24.829073 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.829033    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerStarted","Data":"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"}
Jul 28 10:40:24.829568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.829112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerStarted","Data":"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"}
Jul 28 10:40:24.829568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.829203    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:24.829568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.829310    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:24.830430 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.830408    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:40:24.850982 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:24.850929    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podStartSLOduration=4.009474264 podStartE2EDuration="7.850914132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841439868s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:24.849877698 +0000 UTC m=+1982.770581751" watchObservedRunningTime="2026-07-28 10:40:24.850914132 +0000 UTC m=+1982.771618259"
Jul 28 10:40:25.832896 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:25.832858    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:25.833362 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:25.832890    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:40:25.833910 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:25.833889    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:40:26.838407 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:26.838368    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:40:30.837810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:30.837774    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:40:30.838261 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:30.838238    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:40:31.843501 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:31.843470    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:40:31.844198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:31.844167    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:40:40.838371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:40.838332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:40:41.844995 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:41.844955    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:40:50.838354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:50.838316    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:40:51.844923 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:40:51.844885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:00.839148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:00.839109    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:01.844789 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:01.844748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:10.838339 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:10.838302    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:11.844600 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:11.844559    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:20.838500 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:20.838460    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:21.844873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:21.844832    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:30.838803 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:30.838761    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:31.844563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:31.844520    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:33.561560 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:33.561520    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:40.838844 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:40.838810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:41:43.562699 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:43.562660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:41:47.909398 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:47.909319    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:41:47.909917 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:47.909568    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container" containerID="cri-o://21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24" gracePeriod=30
Jul 28 10:41:47.909917 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:47.909631    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kube-rbac-proxy" containerID="cri-o://17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd" gracePeriod=30
Jul 28 10:41:48.165067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:48.164979    2569 generic.go:356] "Generic (PLEG): container finished" podID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerID="17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd" exitCode=2
Jul 28 10:41:48.165067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:48.165051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerDied","Data":"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"}
Jul 28 10:41:51.839533 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:51.839485    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 10:41:52.656498 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.656472    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:41:52.754922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.754839    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert\") pod \"93c2a29e-6569-4bb7-809a-f68046e2ea64\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") "
Jul 28 10:41:52.754922 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.754891    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") pod \"93c2a29e-6569-4bb7-809a-f68046e2ea64\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") "
Jul 28 10:41:52.755269 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.754931    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location\") pod \"93c2a29e-6569-4bb7-809a-f68046e2ea64\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") "
Jul 28 10:41:52.755269 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755080    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config\") pod \"93c2a29e-6569-4bb7-809a-f68046e2ea64\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") "
Jul 28 10:41:52.755269 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755138    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gts2z\" (UniqueName: \"kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z\") pod \"93c2a29e-6569-4bb7-809a-f68046e2ea64\" (UID: \"93c2a29e-6569-4bb7-809a-f68046e2ea64\") "
Jul 28 10:41:52.755269 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755231    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93c2a29e-6569-4bb7-809a-f68046e2ea64" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:41:52.755475 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755269    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93c2a29e-6569-4bb7-809a-f68046e2ea64" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:52.755527 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755484    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:41:52.755527 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755502    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93c2a29e-6569-4bb7-809a-f68046e2ea64-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:41:52.755634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.755538    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93c2a29e-6569-4bb7-809a-f68046e2ea64" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:52.757020 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.756992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93c2a29e-6569-4bb7-809a-f68046e2ea64" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:52.757139 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.757112    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z" (OuterVolumeSpecName: "kube-api-access-gts2z") pod "93c2a29e-6569-4bb7-809a-f68046e2ea64" (UID: "93c2a29e-6569-4bb7-809a-f68046e2ea64"). InnerVolumeSpecName "kube-api-access-gts2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:52.856229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.856200    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:41:52.856229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.856226    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gts2z\" (UniqueName: \"kubernetes.io/projected/93c2a29e-6569-4bb7-809a-f68046e2ea64-kube-api-access-gts2z\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:41:52.856641 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:52.856236    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93c2a29e-6569-4bb7-809a-f68046e2ea64-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:41:53.192303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.192264    2569 generic.go:356] "Generic (PLEG): container finished" podID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerID="21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24" exitCode=0
Jul 28 10:41:53.192509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.192305    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerDied","Data":"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"}
Jul 28 10:41:53.192509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.192325    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"
Jul 28 10:41:53.192509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.192381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g" event={"ID":"93c2a29e-6569-4bb7-809a-f68046e2ea64","Type":"ContainerDied","Data":"72d0f66de932a9acc0e09679e498d65e0c06a0b15cff23de95bd6c85f48e95ee"}
Jul 28 10:41:53.192509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.192422    2569 scope.go:117] "RemoveContainer" containerID="17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"
Jul 28 10:41:53.201284 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.201265    2569 scope.go:117] "RemoveContainer" containerID="21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"
Jul 28 10:41:53.208777 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.208756    2569 scope.go:117] "RemoveContainer" containerID="42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"
Jul 28 10:41:53.214913 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.214888    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:41:53.216435 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.216418    2569 scope.go:117] "RemoveContainer" containerID="17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"
Jul 28 10:41:53.216727 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:41:53.216706    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd\": container with ID starting with 17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd not found: ID does not exist" containerID="17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"
Jul 28 10:41:53.216891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.216740    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd"} err="failed to get container status \"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd\": rpc error: code = NotFound desc = could not find container \"17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd\": container with ID starting with 17eca06a4128bcda224453f4749590f6bd8fca1d2eaf90f17078b75967d981dd not found: ID does not exist"
Jul 28 10:41:53.216891 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.216767    2569 scope.go:117] "RemoveContainer" containerID="21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"
Jul 28 10:41:53.217501 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:41:53.217462    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24\": container with ID starting with 21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24 not found: ID does not exist" containerID="21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"
Jul 28 10:41:53.217584 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.217498    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24"} err="failed to get container status \"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24\": rpc error: code = NotFound desc = could not find container \"21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24\": container with ID starting with 21d1a4c6a41f296ab6274dbda90b2d96ca9368c23e8534d70ff5c81ce60d8c24 not found: ID does not exist"
Jul 28 10:41:53.217584 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.217520    2569 scope.go:117] "RemoveContainer" containerID="42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"
Jul 28 10:41:53.217792 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:41:53.217772    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150\": container with ID starting with 42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150 not found: ID does not exist" containerID="42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"
Jul 28 10:41:53.217859 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.217802    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150"} err="failed to get container status \"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150\": rpc error: code = NotFound desc = could not find container \"42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150\": container with ID starting with 42a0beae55430556166bead0ad850fd1b6fd7555a174b3be67d8e6d584f5c150 not found: ID does not exist"
Jul 28 10:41:53.218878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:53.218862    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-85k8g"]
Jul 28 10:41:54.566631 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:41:54.566588    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" path="/var/lib/kubelet/pods/93c2a29e-6569-4bb7-809a-f68046e2ea64/volumes"
Jul 28 10:42:22.557999 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:42:22.557972    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:42:22.561809 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:42:22.561779    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:43:18.103172 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:18.103089    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:43:18.103736 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:18.103419    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" containerID="cri-o://07f1863e19a7db1fa4c3f7359f03efd6fe17d20441d82a9504c449c9cf38948e" gracePeriod=30
Jul 28 10:43:18.103736 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:18.103587    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kube-rbac-proxy" containerID="cri-o://b4d3e592591e2d4447babd0d6a959ddbd148ce11768f9d8b3cb596158625dc74" gracePeriod=30
Jul 28 10:43:18.530779 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:18.530740    2569 generic.go:356] "Generic (PLEG): container finished" podID="69c3c537-4235-48dc-b273-be0ea6cde544" containerID="b4d3e592591e2d4447babd0d6a959ddbd148ce11768f9d8b3cb596158625dc74" exitCode=2
Jul 28 10:43:18.530945 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:18.530813    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerDied","Data":"b4d3e592591e2d4447babd0d6a959ddbd148ce11768f9d8b3cb596158625dc74"}
Jul 28 10:43:20.833426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:20.833388    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 10:43:20.838909 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:20.838886    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:43:22.552114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.552083    2569 generic.go:356] "Generic (PLEG): container finished" podID="69c3c537-4235-48dc-b273-be0ea6cde544" containerID="07f1863e19a7db1fa4c3f7359f03efd6fe17d20441d82a9504c449c9cf38948e" exitCode=0
Jul 28 10:43:22.552571 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.552151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerDied","Data":"07f1863e19a7db1fa4c3f7359f03efd6fe17d20441d82a9504c449c9cf38948e"}
Jul 28 10:43:22.649524 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.649501    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:43:22.726917 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.726885    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config\") pod \"69c3c537-4235-48dc-b273-be0ea6cde544\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") "
Jul 28 10:43:22.727081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.726940    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location\") pod \"69c3c537-4235-48dc-b273-be0ea6cde544\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") "
Jul 28 10:43:22.727081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727058    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert\") pod \"69c3c537-4235-48dc-b273-be0ea6cde544\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") "
Jul 28 10:43:22.727173 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls\") pod \"69c3c537-4235-48dc-b273-be0ea6cde544\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") "
Jul 28 10:43:22.727173 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727160    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfjtd\" (UniqueName: \"kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd\") pod \"69c3c537-4235-48dc-b273-be0ea6cde544\" (UID: \"69c3c537-4235-48dc-b273-be0ea6cde544\") "
Jul 28 10:43:22.727283 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727249    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69c3c537-4235-48dc-b273-be0ea6cde544" (UID: "69c3c537-4235-48dc-b273-be0ea6cde544"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:43:22.727378 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727358    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69c3c537-4235-48dc-b273-be0ea6cde544" (UID: "69c3c537-4235-48dc-b273-be0ea6cde544"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:43:22.727492 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727466    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69c3c537-4235-48dc-b273-be0ea6cde544" (UID: "69c3c537-4235-48dc-b273-be0ea6cde544"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:43:22.727492 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727485    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:43:22.727609 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.727500    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c3c537-4235-48dc-b273-be0ea6cde544-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:43:22.729122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.729097    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd" (OuterVolumeSpecName: "kube-api-access-xfjtd") pod "69c3c537-4235-48dc-b273-be0ea6cde544" (UID: "69c3c537-4235-48dc-b273-be0ea6cde544"). InnerVolumeSpecName "kube-api-access-xfjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:43:22.729244 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.729135    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69c3c537-4235-48dc-b273-be0ea6cde544" (UID: "69c3c537-4235-48dc-b273-be0ea6cde544"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:43:22.828149 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.828061    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69c3c537-4235-48dc-b273-be0ea6cde544-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:43:22.828149 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.828091    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c3c537-4235-48dc-b273-be0ea6cde544-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:43:22.828149 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:22.828100    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjtd\" (UniqueName: \"kubernetes.io/projected/69c3c537-4235-48dc-b273-be0ea6cde544-kube-api-access-xfjtd\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:43:23.368944 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.368905    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:43:23.369231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369218    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369233    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369243    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kube-rbac-proxy"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369249    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kube-rbac-proxy"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369256    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kube-rbac-proxy"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369261    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kube-rbac-proxy"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369267    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="storage-initializer"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369272    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="storage-initializer"
Jul 28 10:43:23.369281 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369280    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369286    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369298    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="storage-initializer"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369303    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="storage-initializer"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369346    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kube-rbac-proxy"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369353    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kserve-container"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369361    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c2a29e-6569-4bb7-809a-f68046e2ea64" containerName="kserve-container"
Jul 28 10:43:23.369521 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.369369    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" containerName="kube-rbac-proxy"
Jul 28 10:43:23.373727 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.373679    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:43:23.373874 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.373801    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.376168 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.376150    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 10:43:23.376304 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.376284    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 10:43:23.377168 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.377149    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.379238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.379220    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 10:43:23.383452 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.383432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:43:23.391299 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.391278    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:43:23.432388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432354    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.432545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432392    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.432545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.432545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.432663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxjk9\" (UniqueName: \"kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.432663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432610    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddgx\" (UniqueName: \"kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.432770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432660    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.432770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432704    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.432770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.432770 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.432755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533254 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.533426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533264    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxjk9\" (UniqueName: \"kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.533426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddgx\" (UniqueName: \"kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.533724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.533724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.533724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533544    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.533724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.533681    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.534091 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.534068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.534192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.534169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.534388 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.534362    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.534473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.534413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.534810 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.534786    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.536076 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.536055    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.536171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.536078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.543751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.543725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxjk9\" (UniqueName: \"kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9\") pod \"isvc-canary-stop-predictor-cf655cf76-fqzl4\" (UID: \"3077219d-2b81-4748-a773-663659273263\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.543912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.543889    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddgx\" (UniqueName: \"kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx\") pod \"isvc-canary-stop-v2-predictor-75fcd5578f-27q6p\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.558652 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.558625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9" event={"ID":"69c3c537-4235-48dc-b273-be0ea6cde544","Type":"ContainerDied","Data":"ac93f361b073ceccabf3346cb3b750a420ec5c98ea5bd64a0f21e4254e780f45"}
Jul 28 10:43:23.558991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.558641    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"
Jul 28 10:43:23.558991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.558705    2569 scope.go:117] "RemoveContainer" containerID="b4d3e592591e2d4447babd0d6a959ddbd148ce11768f9d8b3cb596158625dc74"
Jul 28 10:43:23.567243 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.567223    2569 scope.go:117] "RemoveContainer" containerID="07f1863e19a7db1fa4c3f7359f03efd6fe17d20441d82a9504c449c9cf38948e"
Jul 28 10:43:23.574485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.574468    2569 scope.go:117] "RemoveContainer" containerID="b4041149c323d7809c86e00b84d588e9c8bd9e00a964e0e4aabb3237bca1ebf4"
Jul 28 10:43:23.580237 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.580215    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:43:23.583751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.583730    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-ndnk9"]
Jul 28 10:43:23.685837 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.685787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:23.693190 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.693169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:23.823915 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.823876    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:43:23.826285 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:43:23.826259    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cdd01c_57da_4ab5_9dbd_9c9ef22e755e.slice/crio-e89d40ab8c5110ac8f732bc67f56fff574c8faa4128ce98716d7d64c4e7724a1 WatchSource:0}: Error finding container e89d40ab8c5110ac8f732bc67f56fff574c8faa4128ce98716d7d64c4e7724a1: Status 404 returned error can't find the container with id e89d40ab8c5110ac8f732bc67f56fff574c8faa4128ce98716d7d64c4e7724a1
Jul 28 10:43:23.844742 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:23.844718    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:43:23.847258 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:43:23.847227    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3077219d_2b81_4748_a773_663659273263.slice/crio-e5644343b5bd68fd433cfdf7855cf832cc0bc448b825e76bb81089aefe335818 WatchSource:0}: Error finding container e5644343b5bd68fd433cfdf7855cf832cc0bc448b825e76bb81089aefe335818: Status 404 returned error can't find the container with id e5644343b5bd68fd433cfdf7855cf832cc0bc448b825e76bb81089aefe335818
Jul 28 10:43:24.566371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:24.566341    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c3c537-4235-48dc-b273-be0ea6cde544" path="/var/lib/kubelet/pods/69c3c537-4235-48dc-b273-be0ea6cde544/volumes"
Jul 28 10:43:24.566824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:24.566729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerStarted","Data":"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"}
Jul 28 10:43:24.566824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:24.566776    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerStarted","Data":"e89d40ab8c5110ac8f732bc67f56fff574c8faa4128ce98716d7d64c4e7724a1"}
Jul 28 10:43:24.567359 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:24.567330    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerStarted","Data":"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"}
Jul 28 10:43:24.567473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:24.567380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerStarted","Data":"e5644343b5bd68fd433cfdf7855cf832cc0bc448b825e76bb81089aefe335818"}
Jul 28 10:43:27.584611 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:27.584575    2569 generic.go:356] "Generic (PLEG): container finished" podID="3077219d-2b81-4748-a773-663659273263" containerID="7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e" exitCode=0
Jul 28 10:43:27.585075 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:27.584649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerDied","Data":"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"}
Jul 28 10:43:28.589598 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.589563    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerID="66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551" exitCode=0
Jul 28 10:43:28.590028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.589637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerDied","Data":"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"}
Jul 28 10:43:28.591909 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.591889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerStarted","Data":"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"}
Jul 28 10:43:28.592010 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.591934    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerStarted","Data":"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"}
Jul 28 10:43:28.592212 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.592188    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:28.592340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.592242    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:28.593792 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.593769    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:43:28.629559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:28.629512    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podStartSLOduration=1.9815329400000001 podStartE2EDuration="5.629495805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.647962865s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:28.627442233 +0000 UTC m=+2166.548146284" watchObservedRunningTime="2026-07-28 10:43:28.629495805 +0000 UTC m=+2166.550199854"
Jul 28 10:43:29.598256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:29.598213    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerStarted","Data":"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"}
Jul 28 10:43:29.598772 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:29.598290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerStarted","Data":"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"}
Jul 28 10:43:29.598772 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:29.598500    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:43:29.598913 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:29.598898    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:29.618733 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:29.618654    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podStartSLOduration=2.929220682 podStartE2EDuration="6.618639018s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.689418336s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:29.616987872 +0000 UTC m=+2167.537691923" watchObservedRunningTime="2026-07-28 10:43:29.618639018 +0000 UTC m=+2167.539343067"
Jul 28 10:43:30.602045 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:30.602014    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:30.603289 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:30.603260    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:43:31.610559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:31.607410    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:43:34.603292 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:34.603264    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:43:34.603946 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:34.603919    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:43:36.610371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:36.610342    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:43:36.610834 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:36.610806    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:43:44.604162 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:44.604119    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:43:46.611584 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:46.611538    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:43:54.603890 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:54.603846    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:43:56.610893 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:43:56.610853    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:04.604585 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:04.604548    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:44:06.611372 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:06.611333    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:14.603883 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:14.603839    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:44:16.611442 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:16.611400    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:24.603879 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:24.603837    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:44:26.611588 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:26.611544    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:34.604739 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:34.604699    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:44:36.611174 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:36.611137    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:40.561515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:40.561469    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 10:44:46.611489 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:46.611409    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:44:50.565539 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:50.565509    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:44:53.290864 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.290818    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:44:53.291276 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.291056    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container" containerID="cri-o://1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec" gracePeriod=30
Jul 28 10:44:53.291276 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.291076    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kube-rbac-proxy" containerID="cri-o://f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39" gracePeriod=30
Jul 28 10:44:53.329134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.329104    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:44:53.329378 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.329337    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" containerID="cri-o://9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad" gracePeriod=30
Jul 28 10:44:53.329468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.329373    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kube-rbac-proxy" containerID="cri-o://52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d" gracePeriod=30
Jul 28 10:44:53.946568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.946535    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerID="52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d" exitCode=2
Jul 28 10:44:53.946856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.946605    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerDied","Data":"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"}
Jul 28 10:44:53.948555 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.948530    2569 generic.go:356] "Generic (PLEG): container finished" podID="3077219d-2b81-4748-a773-663659273263" containerID="f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39" exitCode=2
Jul 28 10:44:53.948662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:53.948597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerDied","Data":"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"}
Jul 28 10:44:54.599641 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:54.599597    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 10:44:56.607524 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:56.607476    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 10:44:56.610786 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:56.610755    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 10:44:58.035771 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:58.035735    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3077219d_2b81_4748_a773_663659273263.slice/crio-conmon-1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 10:44:58.274724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.274668    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:44:58.335233 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335199    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location\") pod \"3077219d-2b81-4748-a773-663659273263\" (UID: \"3077219d-2b81-4748-a773-663659273263\") "
Jul 28 10:44:58.335402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxjk9\" (UniqueName: \"kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9\") pod \"3077219d-2b81-4748-a773-663659273263\" (UID: \"3077219d-2b81-4748-a773-663659273263\") "
Jul 28 10:44:58.335402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335324    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls\") pod \"3077219d-2b81-4748-a773-663659273263\" (UID: \"3077219d-2b81-4748-a773-663659273263\") "
Jul 28 10:44:58.335402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335364    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config\") pod \"3077219d-2b81-4748-a773-663659273263\" (UID: \"3077219d-2b81-4748-a773-663659273263\") "
Jul 28 10:44:58.335402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335385    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert\") pod \"3077219d-2b81-4748-a773-663659273263\" (UID: \"3077219d-2b81-4748-a773-663659273263\") "
Jul 28 10:44:58.335605 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335540    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3077219d-2b81-4748-a773-663659273263" (UID: "3077219d-2b81-4748-a773-663659273263"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:58.335764 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335744    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3077219d-2b81-4748-a773-663659273263-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.335878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3077219d-2b81-4748-a773-663659273263" (UID: "3077219d-2b81-4748-a773-663659273263"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:44:58.335878 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.335844    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3077219d-2b81-4748-a773-663659273263" (UID: "3077219d-2b81-4748-a773-663659273263"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:44:58.337907 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.337888    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3077219d-2b81-4748-a773-663659273263" (UID: "3077219d-2b81-4748-a773-663659273263"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:44:58.337975 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.337899    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9" (OuterVolumeSpecName: "kube-api-access-gxjk9") pod "3077219d-2b81-4748-a773-663659273263" (UID: "3077219d-2b81-4748-a773-663659273263"). InnerVolumeSpecName "kube-api-access-gxjk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:58.377302 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.377282    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:44:58.436906 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.436871    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location\") pod \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") "
Jul 28 10:44:58.437060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.436925    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls\") pod \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") "
Jul 28 10:44:58.437060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.436956    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cddgx\" (UniqueName: \"kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx\") pod \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") "
Jul 28 10:44:58.437060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.436997    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config\") pod \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") "
Jul 28 10:44:58.437060 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437047    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert\") pod \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\" (UID: \"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e\") "
Jul 28 10:44:58.437306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437132    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" (UID: "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:44:58.437366 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437350    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxjk9\" (UniqueName: \"kubernetes.io/projected/3077219d-2b81-4748-a773-663659273263-kube-api-access-gxjk9\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.437424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437371    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3077219d-2b81-4748-a773-663659273263-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.437424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437373    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" (UID: "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:44:58.437424 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437386    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.437564 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437426    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3077219d-2b81-4748-a773-663659273263-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.437564 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437437    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.437564 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.437480    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" (UID: "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:44:58.438943 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.438922    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx" (OuterVolumeSpecName: "kube-api-access-cddgx") pod "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" (UID: "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e"). InnerVolumeSpecName "kube-api-access-cddgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:44:58.439008 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.438986    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" (UID: "f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:44:58.537976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.537889    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.537976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.537914    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.537976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.537924    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.537976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.537932    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddgx\" (UniqueName: \"kubernetes.io/projected/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e-kube-api-access-cddgx\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:44:58.971368 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.971335    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerID="9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad" exitCode=0
Jul 28 10:44:58.971589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.971400    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"
Jul 28 10:44:58.971589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.971419    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerDied","Data":"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"}
Jul 28 10:44:58.971589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.971487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p" event={"ID":"f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e","Type":"ContainerDied","Data":"e89d40ab8c5110ac8f732bc67f56fff574c8faa4128ce98716d7d64c4e7724a1"}
Jul 28 10:44:58.971589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.971526    2569 scope.go:117] "RemoveContainer" containerID="52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"
Jul 28 10:44:58.973491 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.973466    2569 generic.go:356] "Generic (PLEG): container finished" podID="3077219d-2b81-4748-a773-663659273263" containerID="1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec" exitCode=0
Jul 28 10:44:58.973614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.973494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerDied","Data":"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"}
Jul 28 10:44:58.973614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.973540    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"
Jul 28 10:44:58.973614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.973549    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4" event={"ID":"3077219d-2b81-4748-a773-663659273263","Type":"ContainerDied","Data":"e5644343b5bd68fd433cfdf7855cf832cc0bc448b825e76bb81089aefe335818"}
Jul 28 10:44:58.979898 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.979882    2569 scope.go:117] "RemoveContainer" containerID="9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"
Jul 28 10:44:58.986927 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.986911    2569 scope.go:117] "RemoveContainer" containerID="66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"
Jul 28 10:44:58.995033 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.994976    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:44:58.997390 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.997360    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-cf655cf76-fqzl4"]
Jul 28 10:44:58.998470 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.998442    2569 scope.go:117] "RemoveContainer" containerID="52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"
Jul 28 10:44:58.998801 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:58.998780    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d\": container with ID starting with 52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d not found: ID does not exist" containerID="52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"
Jul 28 10:44:58.998871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.998807    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d"} err="failed to get container status \"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d\": rpc error: code = NotFound desc = could not find container \"52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d\": container with ID starting with 52275b1c5cb1c0182e5f26ab64c31dec8a6412b1e2becaecdc68d08e8a72073d not found: ID does not exist"
Jul 28 10:44:58.998871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.998823    2569 scope.go:117] "RemoveContainer" containerID="9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"
Jul 28 10:44:58.999056 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:58.999039    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad\": container with ID starting with 9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad not found: ID does not exist" containerID="9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"
Jul 28 10:44:58.999108 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.999060    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad"} err="failed to get container status \"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad\": rpc error: code = NotFound desc = could not find container \"9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad\": container with ID starting with 9f6306dcc455cd55dee9d06204056a81efec12c5ab564b382983b1287486b1ad not found: ID does not exist"
Jul 28 10:44:58.999108 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.999074    2569 scope.go:117] "RemoveContainer" containerID="66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"
Jul 28 10:44:58.999287 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:58.999272    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551\": container with ID starting with 66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551 not found: ID does not exist" containerID="66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"
Jul 28 10:44:58.999333 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.999290    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551"} err="failed to get container status \"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551\": rpc error: code = NotFound desc = could not find container \"66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551\": container with ID starting with 66d00b46cbfad48a398d62da32be1dc3da53f7b1feaaa597377aaaff1ea32551 not found: ID does not exist"
Jul 28 10:44:58.999333 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:58.999301    2569 scope.go:117] "RemoveContainer" containerID="f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"
Jul 28 10:44:59.005237 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.005217    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:44:59.009290 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.009269    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75fcd5578f-27q6p"]
Jul 28 10:44:59.011968 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.011949    2569 scope.go:117] "RemoveContainer" containerID="1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"
Jul 28 10:44:59.018909 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.018894    2569 scope.go:117] "RemoveContainer" containerID="7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"
Jul 28 10:44:59.025756 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.025739    2569 scope.go:117] "RemoveContainer" containerID="f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"
Jul 28 10:44:59.026007 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:59.025986    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39\": container with ID starting with f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39 not found: ID does not exist" containerID="f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"
Jul 28 10:44:59.026070 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.026013    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39"} err="failed to get container status \"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39\": rpc error: code = NotFound desc = could not find container \"f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39\": container with ID starting with f11a531c4cce72583ecab1e65fcfbc772dbb9c8dbc8d3d23f71c742745257f39 not found: ID does not exist"
Jul 28 10:44:59.026070 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.026043    2569 scope.go:117] "RemoveContainer" containerID="1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"
Jul 28 10:44:59.026265 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:59.026248    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec\": container with ID starting with 1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec not found: ID does not exist" containerID="1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"
Jul 28 10:44:59.026306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.026272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec"} err="failed to get container status \"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec\": rpc error: code = NotFound desc = could not find container \"1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec\": container with ID starting with 1a29317f9c7b507408b1463aef425597bc9b8020b63b4ff807e0970c11a0fcec not found: ID does not exist"
Jul 28 10:44:59.026306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.026293    2569 scope.go:117] "RemoveContainer" containerID="7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"
Jul 28 10:44:59.026500 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:44:59.026485    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e\": container with ID starting with 7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e not found: ID does not exist" containerID="7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"
Jul 28 10:44:59.026547 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:44:59.026504    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e"} err="failed to get container status \"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e\": rpc error: code = NotFound desc = could not find container \"7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e\": container with ID starting with 7859bd05cca8282bab23d1bd265f0251a4ea748ea10f861f31d881d69a66d45e not found: ID does not exist"
Jul 28 10:45:00.565657 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:00.565623    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3077219d-2b81-4748-a773-663659273263" path="/var/lib/kubelet/pods/3077219d-2b81-4748-a773-663659273263/volumes"
Jul 28 10:45:00.566147 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:00.566132    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" path="/var/lib/kubelet/pods/f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e/volumes"
Jul 28 10:45:03.603157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603116    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603469    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603481    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603501    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kube-rbac-proxy"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603508    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kube-rbac-proxy"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603520    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="storage-initializer"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603526    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="storage-initializer"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603533    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3077219d-2b81-4748-a773-663659273263" containerName="storage-initializer"
Jul 28 10:45:03.603538 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603538    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3077219d-2b81-4748-a773-663659273263" containerName="storage-initializer"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603544    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603550    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603560    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kube-rbac-proxy"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603567    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kube-rbac-proxy"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603614    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kserve-container"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603622    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kserve-container"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603628    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cdd01c-57da-4ab5-9dbd-9c9ef22e755e" containerName="kube-rbac-proxy"
Jul 28 10:45:03.603815 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.603636    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3077219d-2b81-4748-a773-663659273263" containerName="kube-rbac-proxy"
Jul 28 10:45:03.608441 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.608420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.611069 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.611046    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 10:45:03.611202 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.611102    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:45:03.611202 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.611194    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:45:03.611416 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.611402    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 10:45:03.612146 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.612125    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:45:03.612230 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.612199    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:45:03.612306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.612199    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:45:03.617465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.617446    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:45:03.681289 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.681254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.681466 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.681333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.681466 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.681368    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwnh\" (UniqueName: \"kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.681466 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.681443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.681593 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.681486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782318 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwnh\" (UniqueName: \"kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782669 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.782984 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.782963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.783054 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.783001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.784570 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.784553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.790391 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.790371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwnh\" (UniqueName: \"kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh\") pod \"isvc-lightgbm-predictor-54f85ddf88-vwlvk\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:03.919135 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:03.919100    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:04.048915 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:04.048886    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:45:04.051387 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:45:04.051357    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8575f061_ea01_4efc_9d36_a4fa727851d0.slice/crio-13ff4a017b80b43530290b6a7f49c4ab4ef2e11bd8a3d74dcba0d1cf483cb4e2 WatchSource:0}: Error finding container 13ff4a017b80b43530290b6a7f49c4ab4ef2e11bd8a3d74dcba0d1cf483cb4e2: Status 404 returned error can't find the container with id 13ff4a017b80b43530290b6a7f49c4ab4ef2e11bd8a3d74dcba0d1cf483cb4e2
Jul 28 10:45:05.001157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:05.001125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerStarted","Data":"2f09b2a4172803e9d7134c04a7289c7bff0f5877260ae1c8026cedf8535558af"}
Jul 28 10:45:05.001652 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:05.001178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerStarted","Data":"13ff4a017b80b43530290b6a7f49c4ab4ef2e11bd8a3d74dcba0d1cf483cb4e2"}
Jul 28 10:45:08.014983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:08.014892    2569 generic.go:356] "Generic (PLEG): container finished" podID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerID="2f09b2a4172803e9d7134c04a7289c7bff0f5877260ae1c8026cedf8535558af" exitCode=0
Jul 28 10:45:08.015341 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:08.014975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerDied","Data":"2f09b2a4172803e9d7134c04a7289c7bff0f5877260ae1c8026cedf8535558af"}
Jul 28 10:45:08.016157 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:08.016141    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:45:15.052260 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:15.052221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerStarted","Data":"802ebd6c196d98d6bb368f06f97b25143d24c105f9630c0f4a7bb22421cfff3d"}
Jul 28 10:45:15.052637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:15.052288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerStarted","Data":"19de61c2cc36f564f857e5d0aa92b124e183cddd7f0d8d762e938feeeda15e59"}
Jul 28 10:45:15.052637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:15.052471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:15.074814 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:15.074763    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podStartSLOduration=1.829689677 podStartE2EDuration="12.074748828s" totalImagesPullingTime="6.620346617s" totalInitContainerRuntime="3.624712534s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:45:15.071945975 +0000 UTC m=+2272.992650027" watchObservedRunningTime="2026-07-28 10:45:15.074748828 +0000 UTC m=+2272.995452879"
Jul 28 10:45:16.056194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:16.056166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:16.057274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:16.057251    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:45:17.060252 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:17.060213    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:45:22.069047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:22.069018    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:45:22.069656 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:22.069632    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:45:32.070265 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:32.070226    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:45:42.069974 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:42.069936    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:45:52.070166 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:45:52.070124    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:46:02.069754 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:02.069713    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:46:12.069836 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:12.069790    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:46:22.069673 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:22.069586    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 10:46:32.070862 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:32.070827    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:46:33.987355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:33.987320    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:46:33.987753 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:33.987560    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kube-rbac-proxy" containerID="cri-o://802ebd6c196d98d6bb368f06f97b25143d24c105f9630c0f4a7bb22421cfff3d" gracePeriod=30
Jul 28 10:46:33.987753 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:33.987531    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container" containerID="cri-o://19de61c2cc36f564f857e5d0aa92b124e183cddd7f0d8d762e938feeeda15e59" gracePeriod=30
Jul 28 10:46:34.378871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:34.378766    2569 generic.go:356] "Generic (PLEG): container finished" podID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerID="802ebd6c196d98d6bb368f06f97b25143d24c105f9630c0f4a7bb22421cfff3d" exitCode=2
Jul 28 10:46:34.379026 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:34.378923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerDied","Data":"802ebd6c196d98d6bb368f06f97b25143d24c105f9630c0f4a7bb22421cfff3d"}
Jul 28 10:46:37.060896 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:37.060857    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 10:46:38.396906 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.396876    2569 generic.go:356] "Generic (PLEG): container finished" podID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerID="19de61c2cc36f564f857e5d0aa92b124e183cddd7f0d8d762e938feeeda15e59" exitCode=0
Jul 28 10:46:38.397252 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.396948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerDied","Data":"19de61c2cc36f564f857e5d0aa92b124e183cddd7f0d8d762e938feeeda15e59"}
Jul 28 10:46:38.429528 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.429505    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:46:38.567967 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.567876    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location\") pod \"8575f061-ea01-4efc-9d36-a4fa727851d0\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") "
Jul 28 10:46:38.567967 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.567941    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzwnh\" (UniqueName: \"kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh\") pod \"8575f061-ea01-4efc-9d36-a4fa727851d0\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") "
Jul 28 10:46:38.568185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.567980    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls\") pod \"8575f061-ea01-4efc-9d36-a4fa727851d0\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") "
Jul 28 10:46:38.568185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568015    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config\") pod \"8575f061-ea01-4efc-9d36-a4fa727851d0\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") "
Jul 28 10:46:38.568308 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568202    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert\") pod \"8575f061-ea01-4efc-9d36-a4fa727851d0\" (UID: \"8575f061-ea01-4efc-9d36-a4fa727851d0\") "
Jul 28 10:46:38.568308 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568282    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8575f061-ea01-4efc-9d36-a4fa727851d0" (UID: "8575f061-ea01-4efc-9d36-a4fa727851d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:46:38.568480 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568461    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8575f061-ea01-4efc-9d36-a4fa727851d0-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:46:38.568522 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568462    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8575f061-ea01-4efc-9d36-a4fa727851d0" (UID: "8575f061-ea01-4efc-9d36-a4fa727851d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:46:38.568561 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.568543    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8575f061-ea01-4efc-9d36-a4fa727851d0" (UID: "8575f061-ea01-4efc-9d36-a4fa727851d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:46:38.570016 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.569995    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8575f061-ea01-4efc-9d36-a4fa727851d0" (UID: "8575f061-ea01-4efc-9d36-a4fa727851d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:46:38.570097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.570042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh" (OuterVolumeSpecName: "kube-api-access-zzwnh") pod "8575f061-ea01-4efc-9d36-a4fa727851d0" (UID: "8575f061-ea01-4efc-9d36-a4fa727851d0"). InnerVolumeSpecName "kube-api-access-zzwnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:46:38.669628 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.669586    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwnh\" (UniqueName: \"kubernetes.io/projected/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-api-access-zzwnh\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:46:38.669628 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.669623    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8575f061-ea01-4efc-9d36-a4fa727851d0-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:46:38.669628 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.669633    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:46:38.669871 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:38.669643    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8575f061-ea01-4efc-9d36-a4fa727851d0-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:46:39.155333 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155302    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:46:39.155614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155603    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container"
Jul 28 10:46:39.155663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container"
Jul 28 10:46:39.155663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155626    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="storage-initializer"
Jul 28 10:46:39.155663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155631    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="storage-initializer"
Jul 28 10:46:39.155663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155643    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kube-rbac-proxy"
Jul 28 10:46:39.155663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155649    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kube-rbac-proxy"
Jul 28 10:46:39.155841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155707    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kserve-container"
Jul 28 10:46:39.155841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.155716    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" containerName="kube-rbac-proxy"
Jul 28 10:46:39.159013 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.158997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.161410 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.161387    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 10:46:39.161529 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.161429    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 10:46:39.170742 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.170722    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:46:39.275182 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.275144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.275383 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.275194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.275383 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.275278    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.275383 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.275346    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.275383 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.275381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9npz\" (UniqueName: \"kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.376653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.376622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.376842 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.376668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.376842 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.376773    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.376842 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.376836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.377004 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.376880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9npz\" (UniqueName: \"kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.377004 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:46:39.376948    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 10:46:39.377092 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:46:39.377020    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls podName:d9403cb7-8986-43d7-b18d-f7e500337f64 nodeName:}" failed. No retries permitted until 2026-07-28 10:46:39.876997764 +0000 UTC m=+2357.797701797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 10:46:39.377137 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.377100    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.377398 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.377376    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.377470 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.377455    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.387181 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.387153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9npz\" (UniqueName: \"kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.403089 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.403059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk" event={"ID":"8575f061-ea01-4efc-9d36-a4fa727851d0","Type":"ContainerDied","Data":"13ff4a017b80b43530290b6a7f49c4ab4ef2e11bd8a3d74dcba0d1cf483cb4e2"}
Jul 28 10:46:39.403089 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.403078    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"
Jul 28 10:46:39.403536 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.403154    2569 scope.go:117] "RemoveContainer" containerID="802ebd6c196d98d6bb368f06f97b25143d24c105f9630c0f4a7bb22421cfff3d"
Jul 28 10:46:39.411631 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.411615    2569 scope.go:117] "RemoveContainer" containerID="19de61c2cc36f564f857e5d0aa92b124e183cddd7f0d8d762e938feeeda15e59"
Jul 28 10:46:39.419121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.419097    2569 scope.go:117] "RemoveContainer" containerID="2f09b2a4172803e9d7134c04a7289c7bff0f5877260ae1c8026cedf8535558af"
Jul 28 10:46:39.425804 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.425781    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:46:39.431380 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.431358    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-vwlvk"]
Jul 28 10:46:39.880993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.880905    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:39.883377 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:39.883342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:40.068998 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:40.068964    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:40.207356 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:40.207328    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:46:40.209674 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:46:40.209648    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9403cb7_8986_43d7_b18d_f7e500337f64.slice/crio-30a324afaa1140d211f7b72a73e5ea9268aaeb6cce3b102599f0ac167cff6047 WatchSource:0}: Error finding container 30a324afaa1140d211f7b72a73e5ea9268aaeb6cce3b102599f0ac167cff6047: Status 404 returned error can't find the container with id 30a324afaa1140d211f7b72a73e5ea9268aaeb6cce3b102599f0ac167cff6047
Jul 28 10:46:40.409400 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:40.409316    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerStarted","Data":"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"}
Jul 28 10:46:40.409400 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:40.409372    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerStarted","Data":"30a324afaa1140d211f7b72a73e5ea9268aaeb6cce3b102599f0ac167cff6047"}
Jul 28 10:46:40.567674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:40.567630    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8575f061-ea01-4efc-9d36-a4fa727851d0" path="/var/lib/kubelet/pods/8575f061-ea01-4efc-9d36-a4fa727851d0/volumes"
Jul 28 10:46:44.426199 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:44.426165    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerID="6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519" exitCode=0
Jul 28 10:46:44.426622 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:44.426241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerDied","Data":"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"}
Jul 28 10:46:45.432334 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:45.432292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerStarted","Data":"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"}
Jul 28 10:46:45.432766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:45.432380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerStarted","Data":"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"}
Jul 28 10:46:45.432766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:45.432629    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:45.456310 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:45.456261    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podStartSLOduration=2.838836483 podStartE2EDuration="6.45624679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.617410307s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:46:45.454179967 +0000 UTC m=+2363.374884019" watchObservedRunningTime="2026-07-28 10:46:45.45624679 +0000 UTC m=+2363.376950842"
Jul 28 10:46:46.436604 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:46.436573    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:46.437845 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:46.437818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:46:47.440035 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:47.439997    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:46:52.444962 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:52.444932    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:46:52.445622 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:46:52.445592    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:02.446364 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:02.446320    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:12.446055 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:12.446011    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:22.446351 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:22.446314    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:22.588582 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:22.588550    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:47:22.594667 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:22.594647    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:47:32.446347 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:32.446310    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:42.445841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:42.445800    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:47:52.445977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:47:52.445889    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:48:02.446704 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:02.446660    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:48:09.676303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:09.676271    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:48:09.676831 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:09.676484    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" containerID="cri-o://d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386" gracePeriod=30
Jul 28 10:48:09.676831 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:09.676541    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kube-rbac-proxy" containerID="cri-o://5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904" gracePeriod=30
Jul 28 10:48:10.771316 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:10.771284    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerID="5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904" exitCode=2
Jul 28 10:48:10.771697 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:10.771346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerDied","Data":"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"}
Jul 28 10:48:12.440798 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:12.440755    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 10:48:12.446289 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:12.446262    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 10:48:14.714181 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.714153    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:48:14.789306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.789263    2569 generic.go:356] "Generic (PLEG): container finished" podID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerID="d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386" exitCode=0
Jul 28 10:48:14.789306 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.789306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerDied","Data":"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"}
Jul 28 10:48:14.789513 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.789332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"
Jul 28 10:48:14.789513 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.789367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt" event={"ID":"d9403cb7-8986-43d7-b18d-f7e500337f64","Type":"ContainerDied","Data":"30a324afaa1140d211f7b72a73e5ea9268aaeb6cce3b102599f0ac167cff6047"}
Jul 28 10:48:14.789513 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.789407    2569 scope.go:117] "RemoveContainer" containerID="5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"
Jul 28 10:48:14.797309 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.797280    2569 scope.go:117] "RemoveContainer" containerID="d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"
Jul 28 10:48:14.804514 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.804494    2569 scope.go:117] "RemoveContainer" containerID="6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"
Jul 28 10:48:14.813545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.813526    2569 scope.go:117] "RemoveContainer" containerID="5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"
Jul 28 10:48:14.813902 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:48:14.813880    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904\": container with ID starting with 5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904 not found: ID does not exist" containerID="5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"
Jul 28 10:48:14.813983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.813915    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904"} err="failed to get container status \"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904\": rpc error: code = NotFound desc = could not find container \"5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904\": container with ID starting with 5af3d7399cfc50e54b3ee94ce3d7004a7902314cd3257203053d0280adb43904 not found: ID does not exist"
Jul 28 10:48:14.813983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.813940    2569 scope.go:117] "RemoveContainer" containerID="d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"
Jul 28 10:48:14.814194 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:48:14.814176    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386\": container with ID starting with d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386 not found: ID does not exist" containerID="d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"
Jul 28 10:48:14.814241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.814203    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386"} err="failed to get container status \"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386\": rpc error: code = NotFound desc = could not find container \"d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386\": container with ID starting with d7ae299b9be970abcd7a775a9ed6037c93cd55bedd7ea68ec161bc58a790a386 not found: ID does not exist"
Jul 28 10:48:14.814241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.814222    2569 scope.go:117] "RemoveContainer" containerID="6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"
Jul 28 10:48:14.814446 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:48:14.814429    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519\": container with ID starting with 6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519 not found: ID does not exist" containerID="6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"
Jul 28 10:48:14.814497 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.814455    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519"} err="failed to get container status \"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519\": rpc error: code = NotFound desc = could not find container \"6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519\": container with ID starting with 6aa5168a01dc670447c81074de6de0b699f80df57b0bf0bc6dd10b705d5d3519 not found: ID does not exist"
Jul 28 10:48:14.825321 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825297    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:48:14.825627 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825613    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="storage-initializer"
Jul 28 10:48:14.825722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825630    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="storage-initializer"
Jul 28 10:48:14.825722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825654    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container"
Jul 28 10:48:14.825722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825663    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container"
Jul 28 10:48:14.825722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825678    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kube-rbac-proxy"
Jul 28 10:48:14.825722 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825702    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kube-rbac-proxy"
Jul 28 10:48:14.825985 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825777    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kserve-container"
Jul 28 10:48:14.825985 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.825791    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" containerName="kube-rbac-proxy"
Jul 28 10:48:14.830336 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.830319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.832715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.832694    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 10:48:14.832825 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.832723    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 10:48:14.839371 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.839348    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:48:14.859381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859303    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config\") pod \"d9403cb7-8986-43d7-b18d-f7e500337f64\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") "
Jul 28 10:48:14.859381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859334    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9npz\" (UniqueName: \"kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz\") pod \"d9403cb7-8986-43d7-b18d-f7e500337f64\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") "
Jul 28 10:48:14.859381 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859376    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") pod \"d9403cb7-8986-43d7-b18d-f7e500337f64\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") "
Jul 28 10:48:14.859654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859407    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location\") pod \"d9403cb7-8986-43d7-b18d-f7e500337f64\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") "
Jul 28 10:48:14.859654 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859423    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert\") pod \"d9403cb7-8986-43d7-b18d-f7e500337f64\" (UID: \"d9403cb7-8986-43d7-b18d-f7e500337f64\") "
Jul 28 10:48:14.859785 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859742    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9403cb7-8986-43d7-b18d-f7e500337f64" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:48:14.859824 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859790    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9403cb7-8986-43d7-b18d-f7e500337f64" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:48:14.859901 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.859870    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9403cb7-8986-43d7-b18d-f7e500337f64" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:48:14.861481 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.861458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9403cb7-8986-43d7-b18d-f7e500337f64" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:48:14.861573 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.861497    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz" (OuterVolumeSpecName: "kube-api-access-d9npz") pod "d9403cb7-8986-43d7-b18d-f7e500337f64" (UID: "d9403cb7-8986-43d7-b18d-f7e500337f64"). InnerVolumeSpecName "kube-api-access-d9npz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:48:14.960980 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.960943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.961148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.960984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7r2m\" (UniqueName: \"kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.961148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.961148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.961148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961112    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:14.961296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961151    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:48:14.961296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961164    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9npz\" (UniqueName: \"kubernetes.io/projected/d9403cb7-8986-43d7-b18d-f7e500337f64-kube-api-access-d9npz\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:48:14.961296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961175    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9403cb7-8986-43d7-b18d-f7e500337f64-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:48:14.961296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961184    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9403cb7-8986-43d7-b18d-f7e500337f64-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:48:14.961296 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:14.961192    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9403cb7-8986-43d7-b18d-f7e500337f64-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:48:15.062057 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062016    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7r2m\" (UniqueName: \"kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062222    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062528 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062912 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.062993 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.062893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.064741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.064717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.070503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.070480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7r2m\" (UniqueName: \"kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.112839 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.112775    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:48:15.116659 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.116636    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-4kldt"]
Jul 28 10:48:15.141155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.141130    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:48:15.270664 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.270613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:48:15.273201 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:48:15.273176    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3a39de_d88d_4c60_806e_f268fd77008d.slice/crio-44deaac4b35a43f2cd35f5dfebaad78081606f6b608728033fcfa9ba31ccd6e0 WatchSource:0}: Error finding container 44deaac4b35a43f2cd35f5dfebaad78081606f6b608728033fcfa9ba31ccd6e0: Status 404 returned error can't find the container with id 44deaac4b35a43f2cd35f5dfebaad78081606f6b608728033fcfa9ba31ccd6e0
Jul 28 10:48:15.795110 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.795062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerStarted","Data":"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"}
Jul 28 10:48:15.795557 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:15.795138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerStarted","Data":"44deaac4b35a43f2cd35f5dfebaad78081606f6b608728033fcfa9ba31ccd6e0"}
Jul 28 10:48:16.566820 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:16.566787    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9403cb7-8986-43d7-b18d-f7e500337f64" path="/var/lib/kubelet/pods/d9403cb7-8986-43d7-b18d-f7e500337f64/volumes"
Jul 28 10:48:19.813064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:19.813025    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerID="2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126" exitCode=0
Jul 28 10:48:19.813064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:48:19.813067    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerDied","Data":"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"}
Jul 28 10:50:24.398597 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:24.398555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerStarted","Data":"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"}
Jul 28 10:50:24.399036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:24.398626    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerStarted","Data":"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"}
Jul 28 10:50:24.399036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:24.398670    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:50:24.399036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:24.398797    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:50:24.427493 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:24.427440    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" podStartSLOduration=3.207848866 podStartE2EDuration="2m10.427425731s" totalImagesPullingTime="2m3.555240843s" totalInitContainerRuntime="3.664336022s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:50:24.426799646 +0000 UTC m=+2582.347503698" watchObservedRunningTime="2026-07-28 10:50:24.427425731 +0000 UTC m=+2582.348129782"
Jul 28 10:50:30.409320 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:50:30.409289    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:51:00.413766 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:00.413734    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:51:04.999702 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:04.999659    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:51:05.000121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:04.999911    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kserve-container" containerID="cri-o://f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c" gracePeriod=30
Jul 28 10:51:05.000121 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:04.999939    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kube-rbac-proxy" containerID="cri-o://0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2" gracePeriod=30
Jul 28 10:51:05.404627 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:05.404587    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 10:51:05.560698 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:05.560662    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerID="0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2" exitCode=2
Jul 28 10:51:05.560887 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:05.560710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerDied","Data":"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"}
Jul 28 10:51:06.042126 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.042104    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:51:06.065222 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065192    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7r2m\" (UniqueName: \"kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m\") pod \"cd3a39de-d88d-4c60-806e-f268fd77008d\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") "
Jul 28 10:51:06.065417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065277    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert\") pod \"cd3a39de-d88d-4c60-806e-f268fd77008d\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") "
Jul 28 10:51:06.065417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065328    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config\") pod \"cd3a39de-d88d-4c60-806e-f268fd77008d\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") "
Jul 28 10:51:06.065417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065371    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location\") pod \"cd3a39de-d88d-4c60-806e-f268fd77008d\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") "
Jul 28 10:51:06.065417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065396    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls\") pod \"cd3a39de-d88d-4c60-806e-f268fd77008d\" (UID: \"cd3a39de-d88d-4c60-806e-f268fd77008d\") "
Jul 28 10:51:06.066002 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd3a39de-d88d-4c60-806e-f268fd77008d" (UID: "cd3a39de-d88d-4c60-806e-f268fd77008d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:51:06.066165 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.065973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd3a39de-d88d-4c60-806e-f268fd77008d" (UID: "cd3a39de-d88d-4c60-806e-f268fd77008d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:06.066264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.066009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd3a39de-d88d-4c60-806e-f268fd77008d" (UID: "cd3a39de-d88d-4c60-806e-f268fd77008d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:51:06.067960 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.067934    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m" (OuterVolumeSpecName: "kube-api-access-d7r2m") pod "cd3a39de-d88d-4c60-806e-f268fd77008d" (UID: "cd3a39de-d88d-4c60-806e-f268fd77008d"). InnerVolumeSpecName "kube-api-access-d7r2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:51:06.068147 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.068120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd3a39de-d88d-4c60-806e-f268fd77008d" (UID: "cd3a39de-d88d-4c60-806e-f268fd77008d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:51:06.167313 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.167266    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:06.167313 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.167308    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3a39de-d88d-4c60-806e-f268fd77008d-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:06.167313 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.167319    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd3a39de-d88d-4c60-806e-f268fd77008d-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:06.167313 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.167328    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7r2m\" (UniqueName: \"kubernetes.io/projected/cd3a39de-d88d-4c60-806e-f268fd77008d-kube-api-access-d7r2m\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:06.167597 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.167337    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3a39de-d88d-4c60-806e-f268fd77008d-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:06.566755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.566655    2569 generic.go:356] "Generic (PLEG): container finished" podID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerID="f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c" exitCode=0
Jul 28 10:51:06.566755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.566715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerDied","Data":"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"}
Jul 28 10:51:06.566959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.566771    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w" event={"ID":"cd3a39de-d88d-4c60-806e-f268fd77008d","Type":"ContainerDied","Data":"44deaac4b35a43f2cd35f5dfebaad78081606f6b608728033fcfa9ba31ccd6e0"}
Jul 28 10:51:06.566959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.566783    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"
Jul 28 10:51:06.566959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.566803    2569 scope.go:117] "RemoveContainer" containerID="0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"
Jul 28 10:51:06.575161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.575135    2569 scope.go:117] "RemoveContainer" containerID="f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"
Jul 28 10:51:06.583161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.583143    2569 scope.go:117] "RemoveContainer" containerID="2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"
Jul 28 10:51:06.590604 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.590588    2569 scope.go:117] "RemoveContainer" containerID="0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"
Jul 28 10:51:06.590933 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:06.590909    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2\": container with ID starting with 0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2 not found: ID does not exist" containerID="0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"
Jul 28 10:51:06.591038 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.590940    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2"} err="failed to get container status \"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2\": rpc error: code = NotFound desc = could not find container \"0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2\": container with ID starting with 0a3bcc43483327eb09910916a20f54d4fe73240074d9b05d83968ecdb3a0d8e2 not found: ID does not exist"
Jul 28 10:51:06.591038 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.590962    2569 scope.go:117] "RemoveContainer" containerID="f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"
Jul 28 10:51:06.591224 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:06.591205    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c\": container with ID starting with f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c not found: ID does not exist" containerID="f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"
Jul 28 10:51:06.591275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.591230    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c"} err="failed to get container status \"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c\": rpc error: code = NotFound desc = could not find container \"f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c\": container with ID starting with f189c91ba33b9c15d850db502e5053a82b3f727b4f682ce82a97f66905a27e3c not found: ID does not exist"
Jul 28 10:51:06.591275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.591246    2569 scope.go:117] "RemoveContainer" containerID="2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"
Jul 28 10:51:06.591490 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:06.591475    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126\": container with ID starting with 2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126 not found: ID does not exist" containerID="2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"
Jul 28 10:51:06.591547 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.591493    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126"} err="failed to get container status \"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126\": rpc error: code = NotFound desc = could not find container \"2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126\": container with ID starting with 2527d41efa25ab23872bbb80a87399cb53b54dd455f445a3c0be62641281f126 not found: ID does not exist"
Jul 28 10:51:06.592105 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.592088    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:51:06.595419 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:06.595394    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-89v9w"]
Jul 28 10:51:08.566270 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:08.566236    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" path="/var/lib/kubelet/pods/cd3a39de-d88d-4c60-806e-f268fd77008d/volumes"
Jul 28 10:51:10.159985 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.159953    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160256    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kube-rbac-proxy"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160268    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kube-rbac-proxy"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160278    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="storage-initializer"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160284    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="storage-initializer"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160291    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kserve-container"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160297    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kserve-container"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160342    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kserve-container"
Jul 28 10:51:10.160374 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.160352    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3a39de-d88d-4c60-806e-f268fd77008d" containerName="kube-rbac-proxy"
Jul 28 10:51:10.163291 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.163274    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.165926 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.165902    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 10:51:10.166119 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.166105    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:51:10.166333 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.166315    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 10:51:10.166421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.166391    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:51:10.166520 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.166504    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:51:10.167056 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.167042    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:51:10.167179 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.167164    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:51:10.178568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.178544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:10.200856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.200826    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cv6\" (UniqueName: \"kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.200856 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.200857    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.201028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.200875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.201028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.200895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.201028 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.200955    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cv6\" (UniqueName: \"kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302155 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302197    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302405 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:10.302372    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 28 10:51:10.302619 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:10.302441    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls podName:1a86c4a2-4c41-4db4-aaae-bc5546192404 nodeName:}" failed. No retries permitted until 2026-07-28 10:51:10.80242066 +0000 UTC m=+2628.723124692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jul 28 10:51:10.302755 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302733    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302901 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.302947 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.302933    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.313898 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.313872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cv6\" (UniqueName: \"kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.807797 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.807671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:10.810277 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:10.810238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:11.073399 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:11.073312    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:11.200456 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:11.200424    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:11.203472 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:51:11.203443    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a86c4a2_4c41_4db4_aaae_bc5546192404.slice/crio-e618b35de4e5949d0c5f16e9714fd41ed37850fcd256302bb8ccbb4e17f241e6 WatchSource:0}: Error finding container e618b35de4e5949d0c5f16e9714fd41ed37850fcd256302bb8ccbb4e17f241e6: Status 404 returned error can't find the container with id e618b35de4e5949d0c5f16e9714fd41ed37850fcd256302bb8ccbb4e17f241e6
Jul 28 10:51:11.589778 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:11.589743    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerStarted","Data":"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"}
Jul 28 10:51:11.589954 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:11.589799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerStarted","Data":"e618b35de4e5949d0c5f16e9714fd41ed37850fcd256302bb8ccbb4e17f241e6"}
Jul 28 10:51:15.607844 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:15.607812    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerID="5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc" exitCode=0
Jul 28 10:51:15.608238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:15.607882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerDied","Data":"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"}
Jul 28 10:51:16.614670 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.614636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerStarted","Data":"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"}
Jul 28 10:51:16.615185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.614711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerStarted","Data":"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"}
Jul 28 10:51:16.615185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.615041    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:16.615185 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.615166    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:16.616459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.616427    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:51:16.636413 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:16.636353    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podStartSLOduration=2.98534935 podStartE2EDuration="6.636333225s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650983875s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:51:16.632777727 +0000 UTC m=+2634.553481780" watchObservedRunningTime="2026-07-28 10:51:16.636333225 +0000 UTC m=+2634.557037279"
Jul 28 10:51:17.619461 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:17.619415    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:51:18.623843 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:18.623799    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 10:51:23.628275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:23.628240    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:23.629354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:23.629330    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:30.197982 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.197949    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:30.198412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.198199    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container" containerID="cri-o://95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae" gracePeriod=30
Jul 28 10:51:30.198412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.198276    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kube-rbac-proxy" containerID="cri-o://8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583" gracePeriod=30
Jul 28 10:51:30.678481 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.678450    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerID="8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583" exitCode=2
Jul 28 10:51:30.678707 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.678486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerDied","Data":"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"}
Jul 28 10:51:30.936096 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:30.936040    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:31.082473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location\") pod \"1a86c4a2-4c41-4db4-aaae-bc5546192404\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") "
Jul 28 10:51:31.082473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082479    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") pod \"1a86c4a2-4c41-4db4-aaae-bc5546192404\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") "
Jul 28 10:51:31.082741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082503    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8cv6\" (UniqueName: \"kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6\") pod \"1a86c4a2-4c41-4db4-aaae-bc5546192404\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") "
Jul 28 10:51:31.082741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082543    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert\") pod \"1a86c4a2-4c41-4db4-aaae-bc5546192404\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") "
Jul 28 10:51:31.082741 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config\") pod \"1a86c4a2-4c41-4db4-aaae-bc5546192404\" (UID: \"1a86c4a2-4c41-4db4-aaae-bc5546192404\") "
Jul 28 10:51:31.082960 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082933    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a86c4a2-4c41-4db4-aaae-bc5546192404" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:51:31.083027 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.082949    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a86c4a2-4c41-4db4-aaae-bc5546192404" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:51:31.083074 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.083023    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a86c4a2-4c41-4db4-aaae-bc5546192404" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:51:31.084589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.084572    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a86c4a2-4c41-4db4-aaae-bc5546192404" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:51:31.084728 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.084705    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6" (OuterVolumeSpecName: "kube-api-access-v8cv6") pod "1a86c4a2-4c41-4db4-aaae-bc5546192404" (UID: "1a86c4a2-4c41-4db4-aaae-bc5546192404"). InnerVolumeSpecName "kube-api-access-v8cv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:51:31.183873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.183842    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:31.183873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.183868    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a86c4a2-4c41-4db4-aaae-bc5546192404-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:31.183873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.183877    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a86c4a2-4c41-4db4-aaae-bc5546192404-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:31.184112 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.183887    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8cv6\" (UniqueName: \"kubernetes.io/projected/1a86c4a2-4c41-4db4-aaae-bc5546192404-kube-api-access-v8cv6\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:31.184112 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.183897    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a86c4a2-4c41-4db4-aaae-bc5546192404-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:51:31.684400 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.684359    2569 generic.go:356] "Generic (PLEG): container finished" podID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerID="95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae" exitCode=0
Jul 28 10:51:31.684924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.684411    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"
Jul 28 10:51:31.684924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.684443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerDied","Data":"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"}
Jul 28 10:51:31.684924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.684505    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2" event={"ID":"1a86c4a2-4c41-4db4-aaae-bc5546192404","Type":"ContainerDied","Data":"e618b35de4e5949d0c5f16e9714fd41ed37850fcd256302bb8ccbb4e17f241e6"}
Jul 28 10:51:31.684924 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.684533    2569 scope.go:117] "RemoveContainer" containerID="8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"
Jul 28 10:51:31.693611 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.693334    2569 scope.go:117] "RemoveContainer" containerID="95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"
Jul 28 10:51:31.700675 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.700659    2569 scope.go:117] "RemoveContainer" containerID="5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"
Jul 28 10:51:31.707793 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.707774    2569 scope.go:117] "RemoveContainer" containerID="8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"
Jul 28 10:51:31.708045 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:31.708028    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583\": container with ID starting with 8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583 not found: ID does not exist" containerID="8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"
Jul 28 10:51:31.708114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.708054    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583"} err="failed to get container status \"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583\": rpc error: code = NotFound desc = could not find container \"8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583\": container with ID starting with 8136a9a746026e1e3abf85adb5af8b0c431c57d2f6c2e8be47e87685bcc94583 not found: ID does not exist"
Jul 28 10:51:31.708114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.708072    2569 scope.go:117] "RemoveContainer" containerID="95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"
Jul 28 10:51:31.708377 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:31.708342    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae\": container with ID starting with 95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae not found: ID does not exist" containerID="95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"
Jul 28 10:51:31.708428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.708387    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae"} err="failed to get container status \"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae\": rpc error: code = NotFound desc = could not find container \"95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae\": container with ID starting with 95286e5a847baef263acdfa08a0763e8d71fd96fd43651811d4a6ebf997f3aae not found: ID does not exist"
Jul 28 10:51:31.708428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.708408    2569 scope.go:117] "RemoveContainer" containerID="5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"
Jul 28 10:51:31.708648 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:51:31.708627    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc\": container with ID starting with 5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc not found: ID does not exist" containerID="5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"
Jul 28 10:51:31.708715 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.708653    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc"} err="failed to get container status \"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc\": rpc error: code = NotFound desc = could not find container \"5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc\": container with ID starting with 5ef1db14ccd3c5efdcec522109b088db98e2a159d9738faa0a5cb577c70ce0cc not found: ID does not exist"
Jul 28 10:51:31.709563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.709545    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:31.714739 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:31.714720    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-6pcf2"]
Jul 28 10:51:32.567102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:32.567067    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" path="/var/lib/kubelet/pods/1a86c4a2-4c41-4db4-aaae-bc5546192404/volumes"
Jul 28 10:51:35.381058 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381019    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381341    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kube-rbac-proxy"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381354    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kube-rbac-proxy"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381370    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381377    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381385    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="storage-initializer"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381391    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="storage-initializer"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381456    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kube-rbac-proxy"
Jul 28 10:51:35.381460 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.381465    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a86c4a2-4c41-4db4-aaae-bc5546192404" containerName="kserve-container"
Jul 28 10:51:35.383485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.383469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.386102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386058    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:51:35.386102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386091    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 10:51:35.386344 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386124    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:51:35.386344 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386094    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:51:35.386439 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386423    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 10:51:35.387007 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.386990    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:51:35.387110 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.387084    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:51:35.395568 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.395546    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:51:35.519514 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.519457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.519514 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.519520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hs5\" (UniqueName: \"kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.519805 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.519571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.519805 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.519599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.519805 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.519706    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.620476 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.620437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.620476 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.620475    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hs5\" (UniqueName: \"kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.620739 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.620548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.620739 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.620583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.620866 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.620828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.621217 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.621193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.621389 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.621370    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.621452 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.621395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.623104 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.623084    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.643833 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.643777    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hs5\" (UniqueName: \"kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.693485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.693449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:35.830099 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:35.830073    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:51:35.832371 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:51:35.832344    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a88aab6_e0a9_4538_8f25_53249a4f8dca.slice/crio-838770f5cbeae543a9a80ed1233ca5343bb22febb14cc15e730235e03a29de51 WatchSource:0}: Error finding container 838770f5cbeae543a9a80ed1233ca5343bb22febb14cc15e730235e03a29de51: Status 404 returned error can't find the container with id 838770f5cbeae543a9a80ed1233ca5343bb22febb14cc15e730235e03a29de51
Jul 28 10:51:36.706526 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:36.706492    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerStarted","Data":"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"}
Jul 28 10:51:36.706944 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:36.706552    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerStarted","Data":"838770f5cbeae543a9a80ed1233ca5343bb22febb14cc15e730235e03a29de51"}
Jul 28 10:51:40.723450 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:40.723412    2569 generic.go:356] "Generic (PLEG): container finished" podID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerID="6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574" exitCode=0
Jul 28 10:51:40.723962 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:40.723490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerDied","Data":"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"}
Jul 28 10:51:41.729164 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:41.729128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerStarted","Data":"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"}
Jul 28 10:51:41.729618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:41.729202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerStarted","Data":"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"}
Jul 28 10:51:41.729618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:41.729411    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:41.729618 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:41.729564    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:51:41.752959 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:41.752911    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" podStartSLOduration=2.591097102 podStartE2EDuration="6.752896031s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.161798929s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:51:41.751192092 +0000 UTC m=+2659.671896143" watchObservedRunningTime="2026-07-28 10:51:41.752896031 +0000 UTC m=+2659.673600082"
Jul 28 10:51:47.739827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:51:47.739798    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:52:17.743756 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:17.743631    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:52:22.614239 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:22.614208    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:52:22.623234 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:22.623209    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:52:25.462534 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:25.462498    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:52:25.463059 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:25.462846    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kserve-container" containerID="cri-o://ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23" gracePeriod=30
Jul 28 10:52:25.463059 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:25.462913    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kube-rbac-proxy" containerID="cri-o://3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0" gracePeriod=30
Jul 28 10:52:25.911322 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:25.911291    2569 generic.go:356] "Generic (PLEG): container finished" podID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerID="3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0" exitCode=2
Jul 28 10:52:25.911503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:25.911373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerDied","Data":"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"}
Jul 28 10:52:26.812559 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.812529    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:52:26.917439 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.917402    2569 generic.go:356] "Generic (PLEG): container finished" podID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerID="ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23" exitCode=0
Jul 28 10:52:26.917614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.917438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerDied","Data":"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"}
Jul 28 10:52:26.917614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.917468    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"
Jul 28 10:52:26.917614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.917496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7" event={"ID":"2a88aab6-e0a9-4538-8f25-53249a4f8dca","Type":"ContainerDied","Data":"838770f5cbeae543a9a80ed1233ca5343bb22febb14cc15e730235e03a29de51"}
Jul 28 10:52:26.917614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.917524    2569 scope.go:117] "RemoveContainer" containerID="3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"
Jul 28 10:52:26.925533 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.925516    2569 scope.go:117] "RemoveContainer" containerID="ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"
Jul 28 10:52:26.933039 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.933018    2569 scope.go:117] "RemoveContainer" containerID="6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"
Jul 28 10:52:26.940621 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.940604    2569 scope.go:117] "RemoveContainer" containerID="3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"
Jul 28 10:52:26.940904 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:52:26.940885    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0\": container with ID starting with 3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0 not found: ID does not exist" containerID="3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"
Jul 28 10:52:26.940951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.940914    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0"} err="failed to get container status \"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0\": rpc error: code = NotFound desc = could not find container \"3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0\": container with ID starting with 3b6a5275c0c19673bdba7fac39939e71e269824b9c1cd4c3fc332f8ecf68d0d0 not found: ID does not exist"
Jul 28 10:52:26.940951 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.940933    2569 scope.go:117] "RemoveContainer" containerID="ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"
Jul 28 10:52:26.941171 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:52:26.941152    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23\": container with ID starting with ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23 not found: ID does not exist" containerID="ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"
Jul 28 10:52:26.941208 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.941179    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23"} err="failed to get container status \"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23\": rpc error: code = NotFound desc = could not find container \"ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23\": container with ID starting with ca67140cb3a7cc7ad63fdd08267e8e71f3f7be9038926c3c8d4855b0efb63a23 not found: ID does not exist"
Jul 28 10:52:26.941208 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.941194    2569 scope.go:117] "RemoveContainer" containerID="6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"
Jul 28 10:52:26.941422 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:52:26.941404    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574\": container with ID starting with 6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574 not found: ID does not exist" containerID="6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"
Jul 28 10:52:26.941459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.941429    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574"} err="failed to get container status \"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574\": rpc error: code = NotFound desc = could not find container \"6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574\": container with ID starting with 6d3671493ce5a75c93d41cc32e901efeffac4556bdb2c2934c83814eb00fb574 not found: ID does not exist"
Jul 28 10:52:26.949719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.949701    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92hs5\" (UniqueName: \"kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5\") pod \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") "
Jul 28 10:52:26.949795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.949734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert\") pod \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") "
Jul 28 10:52:26.949795 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.949783    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config\") pod \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") "
Jul 28 10:52:26.949873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.949808    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls\") pod \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") "
Jul 28 10:52:26.949873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.949851    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location\") pod \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\" (UID: \"2a88aab6-e0a9-4538-8f25-53249a4f8dca\") "
Jul 28 10:52:26.950166 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.950138    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a88aab6-e0a9-4538-8f25-53249a4f8dca" (UID: "2a88aab6-e0a9-4538-8f25-53249a4f8dca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:52:26.950166 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.950151    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a88aab6-e0a9-4538-8f25-53249a4f8dca" (UID: "2a88aab6-e0a9-4538-8f25-53249a4f8dca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:52:26.950264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.950207    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a88aab6-e0a9-4538-8f25-53249a4f8dca" (UID: "2a88aab6-e0a9-4538-8f25-53249a4f8dca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:52:26.951800 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.951777    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a88aab6-e0a9-4538-8f25-53249a4f8dca" (UID: "2a88aab6-e0a9-4538-8f25-53249a4f8dca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:52:26.951910 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:26.951797    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5" (OuterVolumeSpecName: "kube-api-access-92hs5") pod "2a88aab6-e0a9-4538-8f25-53249a4f8dca" (UID: "2a88aab6-e0a9-4538-8f25-53249a4f8dca"). InnerVolumeSpecName "kube-api-access-92hs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:52:27.050502 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.050468    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a88aab6-e0a9-4538-8f25-53249a4f8dca-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:52:27.050502 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.050496    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:52:27.050502 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.050506    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92hs5\" (UniqueName: \"kubernetes.io/projected/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-api-access-92hs5\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:52:27.050502 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.050514    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:52:27.050898 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.050523    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a88aab6-e0a9-4538-8f25-53249a4f8dca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:52:27.239459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.239423    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:52:27.245724 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:27.245701    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-8cdb7"]
Jul 28 10:52:28.565552 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:28.565520    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" path="/var/lib/kubelet/pods/2a88aab6-e0a9-4538-8f25-53249a4f8dca/volumes"
Jul 28 10:52:30.637916 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.637879    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kserve-container"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638186    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kserve-container"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638193    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kube-rbac-proxy"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638199    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kube-rbac-proxy"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638207    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="storage-initializer"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638213    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="storage-initializer"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638258    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kube-rbac-proxy"
Jul 28 10:52:30.638303 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.638267    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a88aab6-e0a9-4538-8f25-53249a4f8dca" containerName="kserve-container"
Jul 28 10:52:30.641380 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.641362    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.643765 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.643739    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 10:52:30.644181 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644161    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:52:30.644282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:52:30.644282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644163    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:52:30.644282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644164    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 10:52:30.644282 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644198    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:52:30.644503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.644161    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:52:30.652253 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.652232    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:52:30.782285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.782242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.782494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.782325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9xx\" (UniqueName: \"kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.782494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.782378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.782494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.782411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.782494 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.782443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883364 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9xx\" (UniqueName: \"kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883420    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883485 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.883670 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:52:30.883569    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 10:52:30.883670 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:52:30.883656    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls podName:99625ca3-d73c-4d06-9b2d-da976cf5078f nodeName:}" failed. No retries permitted until 2026-07-28 10:52:31.383634084 +0000 UTC m=+2709.304338117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls") pod "isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 10:52:30.883929 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.883911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.884039 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.884022    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.884093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.884078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:30.891895 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:30.891843    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9xx\" (UniqueName: \"kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:31.386781 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.386746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:31.389107 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.389076    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:31.551546 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.551505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:31.689012 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.688977    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:52:31.691457 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:52:31.691435    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99625ca3_d73c_4d06_9b2d_da976cf5078f.slice/crio-3f57dc039e1ababc109588ccd8ecb1f462509aba004c1571dae94bd90715ca9e WatchSource:0}: Error finding container 3f57dc039e1ababc109588ccd8ecb1f462509aba004c1571dae94bd90715ca9e: Status 404 returned error can't find the container with id 3f57dc039e1ababc109588ccd8ecb1f462509aba004c1571dae94bd90715ca9e
Jul 28 10:52:31.941233 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.941147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerStarted","Data":"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"}
Jul 28 10:52:31.941233 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:31.941215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerStarted","Data":"3f57dc039e1ababc109588ccd8ecb1f462509aba004c1571dae94bd90715ca9e"}
Jul 28 10:52:35.959932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:35.959897    2569 generic.go:356] "Generic (PLEG): container finished" podID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerID="ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512" exitCode=0
Jul 28 10:52:35.960355 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:35.959943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerDied","Data":"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"}
Jul 28 10:52:36.028177 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:36.028158    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:52:36.969382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:36.969348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerStarted","Data":"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"}
Jul 28 10:52:38.982487 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:38.982446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerStarted","Data":"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"}
Jul 28 10:52:38.982976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:38.982524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerStarted","Data":"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"}
Jul 28 10:52:38.982976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:38.982565    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:38.982976 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:38.982601    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:39.005609 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:39.005547    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podStartSLOduration=2.639676733 podStartE2EDuration="9.005524484s" totalImagesPullingTime="2.808450038s" totalInitContainerRuntime="3.557397713s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:52:39.003571689 +0000 UTC m=+2716.924275753" watchObservedRunningTime="2026-07-28 10:52:39.005524484 +0000 UTC m=+2716.926228537"
Jul 28 10:52:39.988719 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:39.988411    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:52:45.997242 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:52:45.997209    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:53:11.552589 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:11.552541    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 10:53:16.002979 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:16.002944    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:53:46.004382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:46.004312    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:53:50.732148 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:50.732115    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:53:50.732746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:50.732432    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" containerID="cri-o://4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92" gracePeriod=30
Jul 28 10:53:50.732746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:50.732417    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container" containerID="cri-o://deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0" gracePeriod=30
Jul 28 10:53:50.732746 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:50.732432    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-agent" containerID="cri-o://c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0" gracePeriod=30
Jul 28 10:53:50.992421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:50.992316    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:53:51.274730 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:51.274624    2569 generic.go:356] "Generic (PLEG): container finished" podID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerID="4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92" exitCode=2
Jul 28 10:53:51.274730 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:51.274674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerDied","Data":"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"}
Jul 28 10:53:53.284988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:53.284899    2569 generic.go:356] "Generic (PLEG): container finished" podID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerID="deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0" exitCode=0
Jul 28 10:53:53.284988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:53.284972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerDied","Data":"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"}
Jul 28 10:53:55.892142 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.892107    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:53:55.895673 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.895656    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.898169 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.898145    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 10:53:55.898302 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.898210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 10:53:55.906940 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.906918    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:53:55.981897 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.981859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.982104 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.981937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vr6\" (UniqueName: \"kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.982104 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.981974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.982104 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.982002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.982104 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.982036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:55.992516 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.992476    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:53:55.998218 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:55.998189    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 10:53:56.082919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.082880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.082938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vr6\" (UniqueName: \"kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.082963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.082984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083097 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.083008    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083277 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:53:56.083197    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 10:53:56.083321 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:53:56.083283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls podName:0ec27b6f-ef98-4cc4-894c-5e07c11b821c nodeName:}" failed. No retries permitted until 2026-07-28 10:53:56.583260073 +0000 UTC m=+2794.503964104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-5ctbr" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 28 10:53:56.083321 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.083309    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083788 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.083767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.083826 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.083788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.091751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.091729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vr6\" (UniqueName: \"kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.586632 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.586592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.588963 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.588936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-5ctbr\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.805396 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.805356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:53:56.935468 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:56.935430    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:53:56.938677 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:53:56.938634    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec27b6f_ef98_4cc4_894c_5e07c11b821c.slice/crio-ab95261131f69acae622e6cf705de6c5c50138c7b30baaf81220632935e32d9a WatchSource:0}: Error finding container ab95261131f69acae622e6cf705de6c5c50138c7b30baaf81220632935e32d9a: Status 404 returned error can't find the container with id ab95261131f69acae622e6cf705de6c5c50138c7b30baaf81220632935e32d9a
Jul 28 10:53:57.301461 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:57.301420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerStarted","Data":"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"}
Jul 28 10:53:57.301676 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:53:57.301491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerStarted","Data":"ab95261131f69acae622e6cf705de6c5c50138c7b30baaf81220632935e32d9a"}
Jul 28 10:54:00.992507 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:00.992463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:54:00.992960 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:00.992600    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:54:02.323537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:02.323502    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerID="63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6" exitCode=0
Jul 28 10:54:02.323947 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:02.323576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerDied","Data":"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"}
Jul 28 10:54:05.992405 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:05.992361    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:54:05.998228 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:05.998189    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 10:54:10.992425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:10.992360    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:54:14.384698 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:14.384649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerStarted","Data":"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"}
Jul 28 10:54:14.385117 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:14.384729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerStarted","Data":"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"}
Jul 28 10:54:14.385117 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:14.385010    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:54:14.405081 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:14.405020    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podStartSLOduration=3.56810275 podStartE2EDuration="19.405001862s" totalImagesPullingTime="11.124314455s" totalInitContainerRuntime="4.712584657s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:54:14.40328095 +0000 UTC m=+2812.323985015" watchObservedRunningTime="2026-07-28 10:54:14.405001862 +0000 UTC m=+2812.325705915"
Jul 28 10:54:15.389227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:15.389200    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:54:15.390234 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:15.390210    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:54:15.992435 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:15.992399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 10:54:15.998016 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:15.997981    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 10:54:15.998132 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:15.998103    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:54:16.393486 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:16.393384    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:54:20.920651 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:20.920626    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:54:21.003551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003469    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config\") pod \"99625ca3-d73c-4d06-9b2d-da976cf5078f\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") "
Jul 28 10:54:21.003551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003506    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc9xx\" (UniqueName: \"kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx\") pod \"99625ca3-d73c-4d06-9b2d-da976cf5078f\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") "
Jul 28 10:54:21.003551 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert\") pod \"99625ca3-d73c-4d06-9b2d-da976cf5078f\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") "
Jul 28 10:54:21.003860 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003674    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location\") pod \"99625ca3-d73c-4d06-9b2d-da976cf5078f\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") "
Jul 28 10:54:21.003860 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003793    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") pod \"99625ca3-d73c-4d06-9b2d-da976cf5078f\" (UID: \"99625ca3-d73c-4d06-9b2d-da976cf5078f\") "
Jul 28 10:54:21.003983 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003921    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99625ca3-d73c-4d06-9b2d-da976cf5078f" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:54:21.004067 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.004038    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99625ca3-d73c-4d06-9b2d-da976cf5078f" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:54:21.004123 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.003935    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99625ca3-d73c-4d06-9b2d-da976cf5078f" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:54:21.005623 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.005599    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx" (OuterVolumeSpecName: "kube-api-access-cc9xx") pod "99625ca3-d73c-4d06-9b2d-da976cf5078f" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f"). InnerVolumeSpecName "kube-api-access-cc9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:54:21.005750 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.005705    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99625ca3-d73c-4d06-9b2d-da976cf5078f" (UID: "99625ca3-d73c-4d06-9b2d-da976cf5078f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:54:21.104735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.104679    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99625ca3-d73c-4d06-9b2d-da976cf5078f-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:54:21.104735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.104727    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:54:21.104735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.104738    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9xx\" (UniqueName: \"kubernetes.io/projected/99625ca3-d73c-4d06-9b2d-da976cf5078f-kube-api-access-cc9xx\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:54:21.104735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.104747    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99625ca3-d73c-4d06-9b2d-da976cf5078f-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:54:21.104988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.104755    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99625ca3-d73c-4d06-9b2d-da976cf5078f-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:54:21.398509 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.398478    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:54:21.399112 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.399083    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:54:21.416377 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.416347    2569 generic.go:356] "Generic (PLEG): container finished" podID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerID="c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0" exitCode=137
Jul 28 10:54:21.416508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.416384    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerDied","Data":"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"}
Jul 28 10:54:21.416508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.416412    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"
Jul 28 10:54:21.416508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.416449    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56" event={"ID":"99625ca3-d73c-4d06-9b2d-da976cf5078f","Type":"ContainerDied","Data":"3f57dc039e1ababc109588ccd8ecb1f462509aba004c1571dae94bd90715ca9e"}
Jul 28 10:54:21.416508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.416488    2569 scope.go:117] "RemoveContainer" containerID="4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"
Jul 28 10:54:21.425630 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.425614    2569 scope.go:117] "RemoveContainer" containerID="c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"
Jul 28 10:54:21.433341 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.433324    2569 scope.go:117] "RemoveContainer" containerID="deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"
Jul 28 10:54:21.438769 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.438746    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:54:21.440933 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.440839    2569 scope.go:117] "RemoveContainer" containerID="ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"
Jul 28 10:54:21.442392 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.442374    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-797b9bb4cf-2sv56"]
Jul 28 10:54:21.447906 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.447889    2569 scope.go:117] "RemoveContainer" containerID="4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"
Jul 28 10:54:21.448146 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:54:21.448128    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92\": container with ID starting with 4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92 not found: ID does not exist" containerID="4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"
Jul 28 10:54:21.448193 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448153    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92"} err="failed to get container status \"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92\": rpc error: code = NotFound desc = could not find container \"4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92\": container with ID starting with 4c1b723894668b1ad0383217a6d635ca4c44b66bb1d0dcfefdfa2ac17ca27f92 not found: ID does not exist"
Jul 28 10:54:21.448193 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448171    2569 scope.go:117] "RemoveContainer" containerID="c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"
Jul 28 10:54:21.448415 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:54:21.448397    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0\": container with ID starting with c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0 not found: ID does not exist" containerID="c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"
Jul 28 10:54:21.448465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448421    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0"} err="failed to get container status \"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0\": rpc error: code = NotFound desc = could not find container \"c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0\": container with ID starting with c5b7f5900b46b6712f00f9d7d61c6afac11278cbf3075106f1a6e33f934186a0 not found: ID does not exist"
Jul 28 10:54:21.448465 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448437    2569 scope.go:117] "RemoveContainer" containerID="deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"
Jul 28 10:54:21.448633 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:54:21.448618    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0\": container with ID starting with deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0 not found: ID does not exist" containerID="deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"
Jul 28 10:54:21.448674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448637    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0"} err="failed to get container status \"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0\": rpc error: code = NotFound desc = could not find container \"deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0\": container with ID starting with deca94e0de89a34812f616d9812f4027a643baaa86ee24af8a79164d152b7ec0 not found: ID does not exist"
Jul 28 10:54:21.448674 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448650    2569 scope.go:117] "RemoveContainer" containerID="ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"
Jul 28 10:54:21.448849 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:54:21.448832    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512\": container with ID starting with ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512 not found: ID does not exist" containerID="ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"
Jul 28 10:54:21.448889 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:21.448854    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512"} err="failed to get container status \"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512\": rpc error: code = NotFound desc = could not find container \"ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512\": container with ID starting with ba0b5ed4a2848365410a3125de69bb1547011e3c8551a08cfbf73d8e1b092512 not found: ID does not exist"
Jul 28 10:54:22.566563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:22.566533    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" path="/var/lib/kubelet/pods/99625ca3-d73c-4d06-9b2d-da976cf5078f/volumes"
Jul 28 10:54:31.399908 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:31.399867    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:54:41.399315 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:41.399275    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:54:51.400085 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:54:51.400040    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 10:55:01.399852 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:01.399820    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:55:07.554116 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:07.554085    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:55:07.554591 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:07.554331    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container" containerID="cri-o://e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a" gracePeriod=30
Jul 28 10:55:07.554591 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:07.554374    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kube-rbac-proxy" containerID="cri-o://2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54" gracePeriod=30
Jul 28 10:55:08.609788 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:08.609752    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerID="2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54" exitCode=2
Jul 28 10:55:08.610257 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:08.609832    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerDied","Data":"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"}
Jul 28 10:55:10.298439 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.298414    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:55:10.400179 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400153    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location\") pod \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") "
Jul 28 10:55:10.400328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config\") pod \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") "
Jul 28 10:55:10.400328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400221    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5vr6\" (UniqueName: \"kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6\") pod \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") "
Jul 28 10:55:10.400328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400244    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert\") pod \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") "
Jul 28 10:55:10.400328 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400309    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") pod \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\" (UID: \"0ec27b6f-ef98-4cc4-894c-5e07c11b821c\") "
Jul 28 10:55:10.400580 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400556    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ec27b6f-ef98-4cc4-894c-5e07c11b821c" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:55:10.400630 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.400612    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ec27b6f-ef98-4cc4-894c-5e07c11b821c" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:55:10.402342 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.402314    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6" (OuterVolumeSpecName: "kube-api-access-m5vr6") pod "0ec27b6f-ef98-4cc4-894c-5e07c11b821c" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c"). InnerVolumeSpecName "kube-api-access-m5vr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:55:10.402342 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.402316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ec27b6f-ef98-4cc4-894c-5e07c11b821c" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:55:10.409919 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.409894    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ec27b6f-ef98-4cc4-894c-5e07c11b821c" (UID: "0ec27b6f-ef98-4cc4-894c-5e07c11b821c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:55:10.501409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.501378    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:55:10.501409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.501409    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:55:10.501569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.501419    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:55:10.501569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.501428    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vr6\" (UniqueName: \"kubernetes.io/projected/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-kube-api-access-m5vr6\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:55:10.501569 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.501437    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ec27b6f-ef98-4cc4-894c-5e07c11b821c-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:55:10.619027 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.618995    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerID="e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a" exitCode=0
Jul 28 10:55:10.619194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.619051    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"
Jul 28 10:55:10.619194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.619085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerDied","Data":"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"}
Jul 28 10:55:10.619194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.619146    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr" event={"ID":"0ec27b6f-ef98-4cc4-894c-5e07c11b821c","Type":"ContainerDied","Data":"ab95261131f69acae622e6cf705de6c5c50138c7b30baaf81220632935e32d9a"}
Jul 28 10:55:10.619194 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.619173    2569 scope.go:117] "RemoveContainer" containerID="2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"
Jul 28 10:55:10.632528 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.632501    2569 scope.go:117] "RemoveContainer" containerID="e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"
Jul 28 10:55:10.637330 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.637308    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:55:10.640154 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.640140    2569 scope.go:117] "RemoveContainer" containerID="63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"
Jul 28 10:55:10.643950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.643931    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-5ctbr"]
Jul 28 10:55:10.647525 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.647510    2569 scope.go:117] "RemoveContainer" containerID="2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"
Jul 28 10:55:10.647833 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:55:10.647814    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54\": container with ID starting with 2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54 not found: ID does not exist" containerID="2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"
Jul 28 10:55:10.647877 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.647842    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54"} err="failed to get container status \"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54\": rpc error: code = NotFound desc = could not find container \"2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54\": container with ID starting with 2dfc9fdbf92ef7a0d2ebc7f008f25d476c778dbf1e5fe46fab95031cbfb1df54 not found: ID does not exist"
Jul 28 10:55:10.647877 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.647861    2569 scope.go:117] "RemoveContainer" containerID="e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"
Jul 28 10:55:10.648115 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:55:10.648098    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a\": container with ID starting with e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a not found: ID does not exist" containerID="e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"
Jul 28 10:55:10.648159 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.648123    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a"} err="failed to get container status \"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a\": rpc error: code = NotFound desc = could not find container \"e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a\": container with ID starting with e196f7a0cb114bacd485ecf4427c7b9af431173de4a54090e96ef6e2c344662a not found: ID does not exist"
Jul 28 10:55:10.648159 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.648139    2569 scope.go:117] "RemoveContainer" containerID="63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"
Jul 28 10:55:10.648365 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:55:10.648350    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6\": container with ID starting with 63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6 not found: ID does not exist" containerID="63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"
Jul 28 10:55:10.648412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:10.648369    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6"} err="failed to get container status \"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6\": rpc error: code = NotFound desc = could not find container \"63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6\": container with ID starting with 63d0c04a446242da92f1221b715c721d714df73b02937604598ed477ebbf23b6 not found: ID does not exist"
Jul 28 10:55:12.565802 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.565761    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" path="/var/lib/kubelet/pods/0ec27b6f-ef98-4cc4-894c-5e07c11b821c/volumes"
Jul 28 10:55:12.726300 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726265    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:55:12.726562 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726550    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726564    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726577    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726583    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726590    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="storage-initializer"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726596    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="storage-initializer"
Jul 28 10:55:12.726608 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726608    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="storage-initializer"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726613    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="storage-initializer"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726619    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-agent"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726625    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-agent"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726632    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726638    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726644    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kube-rbac-proxy"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726649    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kube-rbac-proxy"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726722    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-container"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726731    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kserve-container"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726739    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec27b6f-ef98-4cc4-894c-5e07c11b821c" containerName="kube-rbac-proxy"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726747    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kserve-agent"
Jul 28 10:55:12.726827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.726753    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="99625ca3-d73c-4d06-9b2d-da976cf5078f" containerName="kube-rbac-proxy"
Jul 28 10:55:12.729384 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.729362    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.732100 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732075    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 10:55:12.732100 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:55:12.732275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:55:12.732275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732079    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:55:12.732275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732081    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:55:12.732275 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.732079    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:55:12.733049 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.733035    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 10:55:12.739662 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.739643    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:55:12.819786 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.819673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.819786 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.819732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.819990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.819797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.819990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.819843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkc5\" (UniqueName: \"kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.819990 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.819871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921037 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkc5\" (UniqueName: \"kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921210 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921210 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921210 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921210 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921531 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921853 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.921925 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.921840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.923545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.923525    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:12.929676 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:12.929646    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkc5\" (UniqueName: \"kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f7r8t\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:13.040069 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:13.040032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:13.170635 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:13.170474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:55:13.172733 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:55:13.172708    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26236ac5_0aa6_4b2e_b7e9_14c60e657ddf.slice/crio-8408b3cad2ead598c8299913cdb8967da7f97c629036ef19cbb4cc4800e0971f WatchSource:0}: Error finding container 8408b3cad2ead598c8299913cdb8967da7f97c629036ef19cbb4cc4800e0971f: Status 404 returned error can't find the container with id 8408b3cad2ead598c8299913cdb8967da7f97c629036ef19cbb4cc4800e0971f
Jul 28 10:55:13.634661 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:13.634618    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerStarted","Data":"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"}
Jul 28 10:55:13.635066 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:13.634720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerStarted","Data":"8408b3cad2ead598c8299913cdb8967da7f97c629036ef19cbb4cc4800e0971f"}
Jul 28 10:55:17.651937 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:17.651904    2569 generic.go:356] "Generic (PLEG): container finished" podID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerID="a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2" exitCode=0
Jul 28 10:55:17.652299 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:17.651976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerDied","Data":"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"}
Jul 28 10:55:18.657909 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:18.657879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerStarted","Data":"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"}
Jul 28 10:55:18.658308 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:18.657937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerStarted","Data":"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"}
Jul 28 10:55:18.658308 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:18.658123    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:18.681341 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:18.681296    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podStartSLOduration=2.3808733650000002 podStartE2EDuration="6.681281027s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.300407662s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:55:18.680399007 +0000 UTC m=+2876.601103062" watchObservedRunningTime="2026-07-28 10:55:18.681281027 +0000 UTC m=+2876.601985424"
Jul 28 10:55:19.662241 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:19.662210    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:19.663271 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:19.663243    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:55:20.666553 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:20.666513    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:55:25.670904 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:25.670871    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:55:25.674140 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:25.671408    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:55:35.672274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:35.672226    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:55:45.671528 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:45.671483    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:55:55.672222 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:55:55.672182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:56:05.671842 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:05.671811    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:56:14.361238 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:14.361204    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:56:14.361710 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:14.361437    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" containerID="cri-o://c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916" gracePeriod=30
Jul 28 10:56:14.361710 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:14.361465    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kube-rbac-proxy" containerID="cri-o://e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b" gracePeriod=30
Jul 28 10:56:14.881854 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:14.881822    2569 generic.go:356] "Generic (PLEG): container finished" podID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerID="e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b" exitCode=2
Jul 28 10:56:14.882036 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:14.881894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerDied","Data":"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"}
Jul 28 10:56:15.667094 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:15.667049    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 10:56:15.671848 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:15.671827    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 10:56:17.105285 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.105259    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:56:17.255515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255361    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location\") pod \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") "
Jul 28 10:56:17.255515 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert\") pod \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") "
Jul 28 10:56:17.255779 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255554    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config\") pod \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") "
Jul 28 10:56:17.255779 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls\") pod \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") "
Jul 28 10:56:17.255779 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255652    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vkc5\" (UniqueName: \"kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5\") pod \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\" (UID: \"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf\") "
Jul 28 10:56:17.255932 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255905    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" (UID: "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:56:17.255991 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.255964    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" (UID: "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:56:17.257709 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.257668    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5" (OuterVolumeSpecName: "kube-api-access-9vkc5") pod "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" (UID: "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf"). InnerVolumeSpecName "kube-api-access-9vkc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:56:17.257882 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.257863    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" (UID: "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:56:17.265422 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.265398    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" (UID: "26236ac5-0aa6-4b2e-b7e9-14c60e657ddf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:56:17.356385 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.356351    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:56:17.356385 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.356380    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:56:17.356385 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.356390    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:56:17.356617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.356399    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:56:17.356617 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.356409    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vkc5\" (UniqueName: \"kubernetes.io/projected/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf-kube-api-access-9vkc5\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:56:17.895838 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.895798    2569 generic.go:356] "Generic (PLEG): container finished" podID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerID="c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916" exitCode=0
Jul 28 10:56:17.896018 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.895860    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"
Jul 28 10:56:17.896018 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.895874    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerDied","Data":"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"}
Jul 28 10:56:17.896018 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.895938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t" event={"ID":"26236ac5-0aa6-4b2e-b7e9-14c60e657ddf","Type":"ContainerDied","Data":"8408b3cad2ead598c8299913cdb8967da7f97c629036ef19cbb4cc4800e0971f"}
Jul 28 10:56:17.896018 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.895965    2569 scope.go:117] "RemoveContainer" containerID="e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"
Jul 28 10:56:17.904218 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.904196    2569 scope.go:117] "RemoveContainer" containerID="c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"
Jul 28 10:56:17.911382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.911364    2569 scope.go:117] "RemoveContainer" containerID="a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"
Jul 28 10:56:17.917611 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.917590    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:56:17.918752 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.918739    2569 scope.go:117] "RemoveContainer" containerID="e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"
Jul 28 10:56:17.918995 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:56:17.918976    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b\": container with ID starting with e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b not found: ID does not exist" containerID="e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"
Jul 28 10:56:17.919042 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.919003    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b"} err="failed to get container status \"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b\": rpc error: code = NotFound desc = could not find container \"e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b\": container with ID starting with e93dbff66611dcbbfcf1ba5858e9ae70d53deef2d87486def33b31692fea1b3b not found: ID does not exist"
Jul 28 10:56:17.919042 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.919018    2569 scope.go:117] "RemoveContainer" containerID="c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"
Jul 28 10:56:17.919222 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:56:17.919206    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916\": container with ID starting with c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916 not found: ID does not exist" containerID="c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"
Jul 28 10:56:17.919263 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.919225    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916"} err="failed to get container status \"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916\": rpc error: code = NotFound desc = could not find container \"c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916\": container with ID starting with c98926ca0073579de965f8bfe93e9dbf0ba6592c7f78b4f7548033e5e5b57916 not found: ID does not exist"
Jul 28 10:56:17.919263 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.919237    2569 scope.go:117] "RemoveContainer" containerID="a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"
Jul 28 10:56:17.919433 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:56:17.919416    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2\": container with ID starting with a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2 not found: ID does not exist" containerID="a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"
Jul 28 10:56:17.919473 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.919439    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2"} err="failed to get container status \"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2\": rpc error: code = NotFound desc = could not find container \"a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2\": container with ID starting with a6cfd9eab9e547b624155e3a559ab9b15ce96af36a5539097a9329bd3fa27bd2 not found: ID does not exist"
Jul 28 10:56:17.921480 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:17.921462    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f7r8t"]
Jul 28 10:56:18.566412 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:18.566380    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" path="/var/lib/kubelet/pods/26236ac5-0aa6-4b2e-b7e9-14c60e657ddf/volumes"
Jul 28 10:56:19.541917 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.541877    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:56:19.542256 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542239    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container"
Jul 28 10:56:19.542340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542259    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container"
Jul 28 10:56:19.542340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542270    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kube-rbac-proxy"
Jul 28 10:56:19.542340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542278    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kube-rbac-proxy"
Jul 28 10:56:19.542340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542303    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="storage-initializer"
Jul 28 10:56:19.542340 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542313    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="storage-initializer"
Jul 28 10:56:19.542596 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542392    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kserve-container"
Jul 28 10:56:19.542596 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.542403    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26236ac5-0aa6-4b2e-b7e9-14c60e657ddf" containerName="kube-rbac-proxy"
Jul 28 10:56:19.547046 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.547024    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.549556 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.549538    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:56:19.549695 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.549564    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:56:19.549841 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.549828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:56:19.550537 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.550522    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 10:56:19.550582 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.550560    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:56:19.550648 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.550634    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:56:19.550735 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.550634    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 10:56:19.555671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.555651    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:56:19.673122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.673089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.673122 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.673126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.673615 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.673159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.673615 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.673204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdln\" (UniqueName: \"kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.673615 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.673265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774571 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.774535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774571 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.774582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.774618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774827 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.774654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774957 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.774809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdln\" (UniqueName: \"kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.774957 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:56:19.774896    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 10:56:19.775054 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:56:19.774979    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls podName:9f2e7ce6-d988-4c52-81b1-576eead598f0 nodeName:}" failed. No retries permitted until 2026-07-28 10:56:20.274954762 +0000 UTC m=+2938.195658795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 10:56:19.775117 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.775072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.775345 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.775328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.775402 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.775349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:19.783134 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:19.783116    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdln\" (UniqueName: \"kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:20.279171 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.279132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:20.281470 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.281442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:20.458098 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.458044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:20.592212 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.592178    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:56:20.594985 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:56:20.594958    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2e7ce6_d988_4c52_81b1_576eead598f0.slice/crio-6280055f4678e2e07965e45bc17566012ddd0235fc1c4727a53d8901ca8bc9f6 WatchSource:0}: Error finding container 6280055f4678e2e07965e45bc17566012ddd0235fc1c4727a53d8901ca8bc9f6: Status 404 returned error can't find the container with id 6280055f4678e2e07965e45bc17566012ddd0235fc1c4727a53d8901ca8bc9f6
Jul 28 10:56:20.911544 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.911507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerStarted","Data":"52819090858c0b82c976012dc13982f43ad28d50b6c15e2e812308879aa3cd27"}
Jul 28 10:56:20.912055 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:20.911612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerStarted","Data":"6280055f4678e2e07965e45bc17566012ddd0235fc1c4727a53d8901ca8bc9f6"}
Jul 28 10:56:25.933262 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:25.933228    2569 generic.go:356] "Generic (PLEG): container finished" podID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerID="52819090858c0b82c976012dc13982f43ad28d50b6c15e2e812308879aa3cd27" exitCode=0
Jul 28 10:56:25.933703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:25.933299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerDied","Data":"52819090858c0b82c976012dc13982f43ad28d50b6c15e2e812308879aa3cd27"}
Jul 28 10:56:26.943718 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:26.943666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerStarted","Data":"1df6e4ffe53ac27a0d16d0c7aa55c4b27a7f417468529be41c47778132c5da53"}
Jul 28 10:56:26.944274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:26.943787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerStarted","Data":"7151da76750a74c4121de9222c01a3f4f569cfd0be896e1524c2c43a9798607c"}
Jul 28 10:56:26.944274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:26.944014    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:26.962903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:26.962851    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podStartSLOduration=3.398686883 podStartE2EDuration="7.962836364s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.564149481s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:56:26.961850138 +0000 UTC m=+2944.882554189" watchObservedRunningTime="2026-07-28 10:56:26.962836364 +0000 UTC m=+2944.883540419"
Jul 28 10:56:27.948064 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:27.948032    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:27.949260 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:27.949234    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:56:28.951988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:28.951942    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:56:33.956712 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:33.956667    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:56:33.957213 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:33.957137    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:56:43.957900 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:43.957858    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:56:53.957430 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:56:53.957388    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:57:03.957588 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:03.957548    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:57:13.958132 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:13.958101    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:57:21.456872 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:21.456835    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:57:21.457382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:21.457082    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" containerID="cri-o://7151da76750a74c4121de9222c01a3f4f569cfd0be896e1524c2c43a9798607c" gracePeriod=30
Jul 28 10:57:21.457382 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:21.457097    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kube-rbac-proxy" containerID="cri-o://1df6e4ffe53ac27a0d16d0c7aa55c4b27a7f417468529be41c47778132c5da53" gracePeriod=30
Jul 28 10:57:22.166071 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:22.166039    2569 generic.go:356] "Generic (PLEG): container finished" podID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerID="1df6e4ffe53ac27a0d16d0c7aa55c4b27a7f417468529be41c47778132c5da53" exitCode=2
Jul 28 10:57:22.166258 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:22.166090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerDied","Data":"1df6e4ffe53ac27a0d16d0c7aa55c4b27a7f417468529be41c47778132c5da53"}
Jul 28 10:57:22.639130 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:22.639047    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:57:22.652248 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:22.652225    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 10:57:23.953035 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:23.952993    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 28 10:57:23.957479 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:23.957452    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 10:57:24.177493 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.177462    2569 generic.go:356] "Generic (PLEG): container finished" podID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerID="7151da76750a74c4121de9222c01a3f4f569cfd0be896e1524c2c43a9798607c" exitCode=0
Jul 28 10:57:24.177645 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.177518    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerDied","Data":"7151da76750a74c4121de9222c01a3f4f569cfd0be896e1524c2c43a9798607c"}
Jul 28 10:57:24.197579 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.197555    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:57:24.317706 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.317599    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert\") pod \"9f2e7ce6-d988-4c52-81b1-576eead598f0\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") "
Jul 28 10:57:24.317706 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.317642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsdln\" (UniqueName: \"kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln\") pod \"9f2e7ce6-d988-4c52-81b1-576eead598f0\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") "
Jul 28 10:57:24.317953 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.317743    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") pod \"9f2e7ce6-d988-4c52-81b1-576eead598f0\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") "
Jul 28 10:57:24.317953 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.317763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location\") pod \"9f2e7ce6-d988-4c52-81b1-576eead598f0\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") "
Jul 28 10:57:24.317953 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.317797    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config\") pod \"9f2e7ce6-d988-4c52-81b1-576eead598f0\" (UID: \"9f2e7ce6-d988-4c52-81b1-576eead598f0\") "
Jul 28 10:57:24.318115 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.318068    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f2e7ce6-d988-4c52-81b1-576eead598f0" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:57:24.318198 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.318175    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f2e7ce6-d988-4c52-81b1-576eead598f0" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:57:24.319873 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.319843    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln" (OuterVolumeSpecName: "kube-api-access-hsdln") pod "9f2e7ce6-d988-4c52-81b1-576eead598f0" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0"). InnerVolumeSpecName "kube-api-access-hsdln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:57:24.320013 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.319938    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f2e7ce6-d988-4c52-81b1-576eead598f0" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:57:24.327792 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.327761    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f2e7ce6-d988-4c52-81b1-576eead598f0" (UID: "9f2e7ce6-d988-4c52-81b1-576eead598f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:57:24.419227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.419178    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:57:24.419227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.419218    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f2e7ce6-d988-4c52-81b1-576eead598f0-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:57:24.419227 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.419229    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdln\" (UniqueName: \"kubernetes.io/projected/9f2e7ce6-d988-4c52-81b1-576eead598f0-kube-api-access-hsdln\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:57:24.419476 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.419250    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2e7ce6-d988-4c52-81b1-576eead598f0-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:57:24.419476 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:24.419259    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f2e7ce6-d988-4c52-81b1-576eead598f0-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:57:25.183503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.183474    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"
Jul 28 10:57:25.183950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.183473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km" event={"ID":"9f2e7ce6-d988-4c52-81b1-576eead598f0","Type":"ContainerDied","Data":"6280055f4678e2e07965e45bc17566012ddd0235fc1c4727a53d8901ca8bc9f6"}
Jul 28 10:57:25.183950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.183614    2569 scope.go:117] "RemoveContainer" containerID="1df6e4ffe53ac27a0d16d0c7aa55c4b27a7f417468529be41c47778132c5da53"
Jul 28 10:57:25.192160 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.192140    2569 scope.go:117] "RemoveContainer" containerID="7151da76750a74c4121de9222c01a3f4f569cfd0be896e1524c2c43a9798607c"
Jul 28 10:57:25.199453 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.199435    2569 scope.go:117] "RemoveContainer" containerID="52819090858c0b82c976012dc13982f43ad28d50b6c15e2e812308879aa3cd27"
Jul 28 10:57:25.206454 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.206430    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:57:25.211463 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:25.211443    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ls9km"]
Jul 28 10:57:26.568274 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.568242    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" path="/var/lib/kubelet/pods/9f2e7ce6-d988-4c52-81b1-576eead598f0/volumes"
Jul 28 10:57:26.630830 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.630799    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:57:26.631114 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631100    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="storage-initializer"
Jul 28 10:57:26.631167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631117    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="storage-initializer"
Jul 28 10:57:26.631167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631129    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container"
Jul 28 10:57:26.631167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631135    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container"
Jul 28 10:57:26.631167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631147    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kube-rbac-proxy"
Jul 28 10:57:26.631167 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631152    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kube-rbac-proxy"
Jul 28 10:57:26.631317 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631196    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kube-rbac-proxy"
Jul 28 10:57:26.631317 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.631203    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2e7ce6-d988-4c52-81b1-576eead598f0" containerName="kserve-container"
Jul 28 10:57:26.635830 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.635815    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.638508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.638489    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 10:57:26.638671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.638522    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:57:26.638671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.638571    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:57:26.638671 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.638636    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:57:26.639503 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.639479    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 10:57:26.639614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.639528    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:57:26.639614 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.639558    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:57:26.643346 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.643325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:57:26.736945 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.736901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.737161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.736970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzps\" (UniqueName: \"kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.737161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.737002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.737161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.737082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.737161 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.737131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.837977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.837884    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.837977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.837922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzps\" (UniqueName: \"kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.837977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.837938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.837977 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.837968    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.838263 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.837996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.838370 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.838347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.838634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.838609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.838714 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.838648    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.840415 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.840397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.845590 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.845569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzps\" (UniqueName: \"kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps\") pod \"isvc-pmml-predictor-8674795f59-8smtv\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:26.946910 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:26.946848    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:27.078637 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:27.078601    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:57:27.081627 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:57:27.081594    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d67e0f7_4575_48d6_a7ee_4ef9b898c4c3.slice/crio-0452ee747e5dd535e6b7300cc665f57435b050c81dd381a1944f1b089f4f90e7 WatchSource:0}: Error finding container 0452ee747e5dd535e6b7300cc665f57435b050c81dd381a1944f1b089f4f90e7: Status 404 returned error can't find the container with id 0452ee747e5dd535e6b7300cc665f57435b050c81dd381a1944f1b089f4f90e7
Jul 28 10:57:27.194444 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:27.194395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerStarted","Data":"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"}
Jul 28 10:57:27.194639 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:27.194463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerStarted","Data":"0452ee747e5dd535e6b7300cc665f57435b050c81dd381a1944f1b089f4f90e7"}
Jul 28 10:57:31.211249 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:31.211158    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerID="e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082" exitCode=0
Jul 28 10:57:31.211249 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:31.211205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerDied","Data":"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"}
Jul 28 10:57:38.247775 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:38.247745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerStarted","Data":"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"}
Jul 28 10:57:39.253606 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:39.253561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerStarted","Data":"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"}
Jul 28 10:57:39.254044 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:39.253717    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:39.273205 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:39.273160    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podStartSLOduration=2.555505701 podStartE2EDuration="13.273144686s" totalImagesPullingTime="6.922915045s" totalInitContainerRuntime="3.79472394s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:57:39.270625136 +0000 UTC m=+3017.191329189" watchObservedRunningTime="2026-07-28 10:57:39.273144686 +0000 UTC m=+3017.193848738"
Jul 28 10:57:40.257785 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:40.257746    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:40.258903 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:40.258877    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:57:41.261425 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:41.261386    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:57:46.266264 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:46.266232    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:57:46.266749 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:46.266723    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:57:56.267545 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:57:56.267501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:06.267421 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:06.267381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:16.267713 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:16.267610    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:26.267681 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:26.267643    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:36.267295 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:36.267251    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:46.267428 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:46.267391    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 10:58:56.267801 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:56.267770    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:58:57.871840 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:57.871805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:58:57.872307 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:57.872077    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container" containerID="cri-o://2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048" gracePeriod=30
Jul 28 10:58:57.872307 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:57.872115    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kube-rbac-proxy" containerID="cri-o://d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43" gracePeriod=30
Jul 28 10:58:58.559417 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:58.559386    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerID="d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43" exitCode=2
Jul 28 10:58:58.559585 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:58:58.559457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerDied","Data":"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"}
Jul 28 10:59:01.262459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.262418    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 28 10:59:01.519794 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.519732    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:59:01.571751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.571718    2569 generic.go:356] "Generic (PLEG): container finished" podID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerID="2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048" exitCode=0
Jul 28 10:59:01.571751 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.571753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerDied","Data":"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"}
Jul 28 10:59:01.571970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.571772    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"
Jul 28 10:59:01.571970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.571797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv" event={"ID":"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3","Type":"ContainerDied","Data":"0452ee747e5dd535e6b7300cc665f57435b050c81dd381a1944f1b089f4f90e7"}
Jul 28 10:59:01.571970 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.571825    2569 scope.go:117] "RemoveContainer" containerID="d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"
Jul 28 10:59:01.579506 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.579487    2569 scope.go:117] "RemoveContainer" containerID="2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"
Jul 28 10:59:01.586556 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.586539    2569 scope.go:117] "RemoveContainer" containerID="e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"
Jul 28 10:59:01.593575 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.593560    2569 scope.go:117] "RemoveContainer" containerID="d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"
Jul 28 10:59:01.593880 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:59:01.593861    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43\": container with ID starting with d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43 not found: ID does not exist" containerID="d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"
Jul 28 10:59:01.593949 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.593888    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43"} err="failed to get container status \"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43\": rpc error: code = NotFound desc = could not find container \"d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43\": container with ID starting with d509236e8f6d4861afa081f2c1bdf57a1b6a16425d9665ab67c526f55b2eef43 not found: ID does not exist"
Jul 28 10:59:01.593949 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.593909    2569 scope.go:117] "RemoveContainer" containerID="2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"
Jul 28 10:59:01.594108 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:59:01.594089    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048\": container with ID starting with 2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048 not found: ID does not exist" containerID="2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"
Jul 28 10:59:01.594152 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.594114    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048"} err="failed to get container status \"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048\": rpc error: code = NotFound desc = could not find container \"2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048\": container with ID starting with 2b2002229e140ae99c35bab6e1b2c35818098056cf0b7bb389c747a5e17a6048 not found: ID does not exist"
Jul 28 10:59:01.594152 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.594128    2569 scope.go:117] "RemoveContainer" containerID="e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"
Jul 28 10:59:01.594335 ip-10-0-133-108 kubenswrapper[2569]: E0728 10:59:01.594316    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082\": container with ID starting with e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082 not found: ID does not exist" containerID="e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"
Jul 28 10:59:01.594376 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.594340    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082"} err="failed to get container status \"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082\": rpc error: code = NotFound desc = could not find container \"e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082\": container with ID starting with e839676fe722598504961369375567028b27a1d748a31bd28cce000a5d6d9082 not found: ID does not exist"
Jul 28 10:59:01.633229 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633200    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config\") pod \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") "
Jul 28 10:59:01.633349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633263    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls\") pod \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") "
Jul 28 10:59:01.633349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633298    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert\") pod \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") "
Jul 28 10:59:01.633349 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633345    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pzps\" (UniqueName: \"kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps\") pod \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") "
Jul 28 10:59:01.633508 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location\") pod \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\" (UID: \"2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3\") "
Jul 28 10:59:01.633789 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" (UID: "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:01.633887 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633767    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" (UID: "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:01.633887 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.633803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" (UID: "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:01.635235 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.635215    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" (UID: "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:59:01.635459 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.635444    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps" (OuterVolumeSpecName: "kube-api-access-9pzps") pod "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" (UID: "2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3"). InnerVolumeSpecName "kube-api-access-9pzps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:59:01.734581 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.734545    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.734581 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.734574    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.734581 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.734585    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzps\" (UniqueName: \"kubernetes.io/projected/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-api-access-9pzps\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.734857 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.734596    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.734857 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.734604    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.894634 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.894600    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:59:01.898409 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:01.898387    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-8smtv"]
Jul 28 10:59:02.565667 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:02.565635    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" path="/var/lib/kubelet/pods/2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3/volumes"
Jul 28 10:59:03.037950 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.037905    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 10:59:03.038315 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038295    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="storage-initializer"
Jul 28 10:59:03.038404 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038316    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="storage-initializer"
Jul 28 10:59:03.038404 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038329    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container"
Jul 28 10:59:03.038404 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038337    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container"
Jul 28 10:59:03.038404 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038359    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kube-rbac-proxy"
Jul 28 10:59:03.038404 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038369    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kube-rbac-proxy"
Jul 28 10:59:03.038703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038439    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kube-rbac-proxy"
Jul 28 10:59:03.038703 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.038456    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d67e0f7-4575-48d6-a7ee-4ef9b898c4c3" containerName="kserve-container"
Jul 28 10:59:03.043129 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.043108    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.045981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.045947    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 10:59:03.045981 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.045965    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 10:59:03.046156 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.045965    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 10:59:03.046156 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.046051    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 10:59:03.046156 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.046073    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 10:59:03.046156 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.045966    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:59:03.046329 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.046316    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:59:03.053205 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.053180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 10:59:03.148577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.148540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzgg\" (UniqueName: \"kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.148577 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.148579    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.148829 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.148712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.148829 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.148754    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.148829 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.148774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249153 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249267    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249354 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzgg\" (UniqueName: \"kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.249761 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.249737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.250047 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.250020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.250192 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.250170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.251948 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.251920    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.258136 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.258115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzgg\" (UniqueName: \"kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg\") pod \"isvc-pmml-runtime-predictor-885fc8777-jv6tm\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.354694 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.354596    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:03.485620 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.485594    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 10:59:03.488188 ip-10-0-133-108 kubenswrapper[2569]: W0728 10:59:03.488159    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7e0c8e_119c_41a7_862f_6034aa2528ac.slice/crio-09cf65b4dd10995790baedb0cb8dbe58061fffd437a28c80265976a19e171122 WatchSource:0}: Error finding container 09cf65b4dd10995790baedb0cb8dbe58061fffd437a28c80265976a19e171122: Status 404 returned error can't find the container with id 09cf65b4dd10995790baedb0cb8dbe58061fffd437a28c80265976a19e171122
Jul 28 10:59:03.583221 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.583178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerStarted","Data":"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"}
Jul 28 10:59:03.583639 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:03.583250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerStarted","Data":"09cf65b4dd10995790baedb0cb8dbe58061fffd437a28c80265976a19e171122"}
Jul 28 10:59:07.600663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:07.600577    2569 generic.go:356] "Generic (PLEG): container finished" podID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerID="0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13" exitCode=0
Jul 28 10:59:07.600663 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:07.600648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerDied","Data":"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"}
Jul 28 10:59:08.606653 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:08.606619    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerStarted","Data":"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"}
Jul 28 10:59:08.607093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:08.606708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerStarted","Data":"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"}
Jul 28 10:59:08.607093 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:08.606901    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:08.625168 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:08.625113    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podStartSLOduration=1.8667370189999999 podStartE2EDuration="5.625093692s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.758356673s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:08.624627753 +0000 UTC m=+3106.545331805" watchObservedRunningTime="2026-07-28 10:59:08.625093692 +0000 UTC m=+3106.545797744"
Jul 28 10:59:09.611164 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:09.611138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:09.612426 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:09.612392    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:10.615231 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:10.615185    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:15.619598 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:15.619567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 10:59:15.620102 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:15.620077    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:25.620988 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:25.620949    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:35.620563 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:35.620519    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:45.620463 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:45.620375    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 10:59:55.620437 ip-10-0-133-108 kubenswrapper[2569]: I0728 10:59:55.620391    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 11:00:05.620977 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:05.620939    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 11:00:15.620260 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:15.620218    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 11:00:25.620729 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:25.620673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 11:00:34.172010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:34.171978    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 11:00:34.174531 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:34.172201    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" containerID="cri-o://df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4" gracePeriod=30
Jul 28 11:00:34.174531 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:34.172246    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kube-rbac-proxy" containerID="cri-o://41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3" gracePeriod=30
Jul 28 11:00:34.956952 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:34.956911    2569 generic.go:356] "Generic (PLEG): container finished" podID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerID="41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3" exitCode=2
Jul 28 11:00:34.957130 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:34.956981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerDied","Data":"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"}
Jul 28 11:00:35.615873 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:35.615834    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 11:00:35.620201 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:35.620179    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 11:00:37.922985 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.922959    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 11:00:37.970144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.970053    2569 generic.go:356] "Generic (PLEG): container finished" podID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerID="df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4" exitCode=0
Jul 28 11:00:37.970144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.970107    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"
Jul 28 11:00:37.970337 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.970140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerDied","Data":"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"}
Jul 28 11:00:37.970337 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.970203    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm" event={"ID":"8a7e0c8e-119c-41a7-862f-6034aa2528ac","Type":"ContainerDied","Data":"09cf65b4dd10995790baedb0cb8dbe58061fffd437a28c80265976a19e171122"}
Jul 28 11:00:37.970337 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.970235    2569 scope.go:117] "RemoveContainer" containerID="41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"
Jul 28 11:00:37.978130 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.978112    2569 scope.go:117] "RemoveContainer" containerID="df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"
Jul 28 11:00:37.985408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.985387    2569 scope.go:117] "RemoveContainer" containerID="0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"
Jul 28 11:00:37.992514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.992496    2569 scope.go:117] "RemoveContainer" containerID="41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"
Jul 28 11:00:37.992768 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:00:37.992746    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3\": container with ID starting with 41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3 not found: ID does not exist" containerID="41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"
Jul 28 11:00:37.992862 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.992776    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3"} err="failed to get container status \"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3\": rpc error: code = NotFound desc = could not find container \"41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3\": container with ID starting with 41647e6fd31c22df2551c3dcaadd43728ec294bc5e51762b7529bd973a50ebe3 not found: ID does not exist"
Jul 28 11:00:37.992862 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.992794    2569 scope.go:117] "RemoveContainer" containerID="df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"
Jul 28 11:00:37.993008 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:00:37.992991    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4\": container with ID starting with df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4 not found: ID does not exist" containerID="df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"
Jul 28 11:00:37.993045 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.993014    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4"} err="failed to get container status \"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4\": rpc error: code = NotFound desc = could not find container \"df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4\": container with ID starting with df20bae332cd15a0468de3f08883b68a3e74fbfbd85abe6f284cd20409036ba4 not found: ID does not exist"
Jul 28 11:00:37.993045 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.993029    2569 scope.go:117] "RemoveContainer" containerID="0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"
Jul 28 11:00:37.993254 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:00:37.993240    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13\": container with ID starting with 0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13 not found: ID does not exist" containerID="0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"
Jul 28 11:00:37.993306 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:37.993255    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13"} err="failed to get container status \"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13\": rpc error: code = NotFound desc = could not find container \"0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13\": container with ID starting with 0403c00ba7dc0e67cc958ad6986456dd345e56221254e0b86bfeaebaa325ba13 not found: ID does not exist"
Jul 28 11:00:38.046212 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046180    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config\") pod \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") "
Jul 28 11:00:38.046353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046246    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location\") pod \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") "
Jul 28 11:00:38.046353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046274    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert\") pod \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") "
Jul 28 11:00:38.046353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046299    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls\") pod \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") "
Jul 28 11:00:38.046353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046323    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gzgg\" (UniqueName: \"kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg\") pod \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\" (UID: \"8a7e0c8e-119c-41a7-862f-6034aa2528ac\") "
Jul 28 11:00:38.046657 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046624    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a7e0c8e-119c-41a7-862f-6034aa2528ac" (UID: "8a7e0c8e-119c-41a7-862f-6034aa2528ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:00:38.046657 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046644    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a7e0c8e-119c-41a7-862f-6034aa2528ac" (UID: "8a7e0c8e-119c-41a7-862f-6034aa2528ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:38.046819 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.046670    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a7e0c8e-119c-41a7-862f-6034aa2528ac" (UID: "8a7e0c8e-119c-41a7-862f-6034aa2528ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:00:38.048349 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.048327    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a7e0c8e-119c-41a7-862f-6034aa2528ac" (UID: "8a7e0c8e-119c-41a7-862f-6034aa2528ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:00:38.048446 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.048385    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg" (OuterVolumeSpecName: "kube-api-access-7gzgg") pod "8a7e0c8e-119c-41a7-862f-6034aa2528ac" (UID: "8a7e0c8e-119c-41a7-862f-6034aa2528ac"). InnerVolumeSpecName "kube-api-access-7gzgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:00:38.147388 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.147356    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.147388 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.147384    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.147388 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.147393    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a7e0c8e-119c-41a7-862f-6034aa2528ac-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.147629 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.147403    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a7e0c8e-119c-41a7-862f-6034aa2528ac-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.147629 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.147411    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzgg\" (UniqueName: \"kubernetes.io/projected/8a7e0c8e-119c-41a7-862f-6034aa2528ac-kube-api-access-7gzgg\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.292557 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.292524    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 11:00:38.298088 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.298062    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-jv6tm"]
Jul 28 11:00:38.566168 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:38.566089    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" path="/var/lib/kubelet/pods/8a7e0c8e-119c-41a7-862f-6034aa2528ac/volumes"
Jul 28 11:00:39.349290 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349254    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349550    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kube-rbac-proxy"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349562    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kube-rbac-proxy"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349578    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="storage-initializer"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349584    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="storage-initializer"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349600    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349605    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349652    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kserve-container"
Jul 28 11:00:39.349676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.349662    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7e0c8e-119c-41a7-862f-6034aa2528ac" containerName="kube-rbac-proxy"
Jul 28 11:00:39.355914 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.355885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.361722 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.361680    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 11:00:39.362137 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.362118    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:00:39.363147 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.363125    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:00:39.363309 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.363268    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:00:39.363450 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.363187    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:00:39.363537 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.363223    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 11:00:39.363611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.363175    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:00:39.365184 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.365161    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:00:39.459744 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.459709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9w7n\" (UniqueName: \"kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.459956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.459772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.459956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.459839    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.459956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.459882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.459956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.459909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.560543 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.560507    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.560762 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.560551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.560762 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.560578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.560762 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.560714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.560762 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.560759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9w7n\" (UniqueName: \"kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.561177 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.561151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.561287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.561267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.561363 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.561335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.563089 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.563068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.568649 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.568631    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9w7n\" (UniqueName: \"kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.666948 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.666906    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:39.797763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.797731    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:00:39.800799 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:00:39.800767    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c3b9a5_3fd5_4554_a3f0_693649c4efb6.slice/crio-c08cdb5956bcb5d46b7b91ab008a12c0de42cdbe1f0ee5021d0751f024975a0d WatchSource:0}: Error finding container c08cdb5956bcb5d46b7b91ab008a12c0de42cdbe1f0ee5021d0751f024975a0d: Status 404 returned error can't find the container with id c08cdb5956bcb5d46b7b91ab008a12c0de42cdbe1f0ee5021d0751f024975a0d
Jul 28 11:00:39.980276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.980179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerStarted","Data":"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"}
Jul 28 11:00:39.980276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:39.980254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerStarted","Data":"c08cdb5956bcb5d46b7b91ab008a12c0de42cdbe1f0ee5021d0751f024975a0d"}
Jul 28 11:00:43.998558 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:43.998522    2569 generic.go:356] "Generic (PLEG): container finished" podID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerID="e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7" exitCode=0
Jul 28 11:00:43.998966 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:43.998578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerDied","Data":"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"}
Jul 28 11:00:45.004653 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:45.004613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerStarted","Data":"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"}
Jul 28 11:00:45.005119 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:45.004673    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerStarted","Data":"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"}
Jul 28 11:00:45.005119 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:45.004879    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:45.024444 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:45.024397    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podStartSLOduration=2.260432944 podStartE2EDuration="6.024383306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763950362s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:45.023706377 +0000 UTC m=+3202.944410431" watchObservedRunningTime="2026-07-28 11:00:45.024383306 +0000 UTC m=+3202.945087357"
Jul 28 11:00:46.012707 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:46.008638    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:46.014055 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:46.014011    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:00:47.012846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:47.012811    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:00:52.017725 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:52.017680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:00:52.018226 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:00:52.018202    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:02.018529 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:02.018492    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:12.019154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:12.019118    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:22.019178 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:22.019085    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:32.018968 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:32.018927    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:42.018963 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:42.018926    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:01:52.018223 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:01:52.018182    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:02:02.019587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:02.019548    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:02:10.476831 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:10.476790    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:02:10.477272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:10.477012    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" containerID="cri-o://400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c" gracePeriod=30
Jul 28 11:02:10.477272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:10.477081    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kube-rbac-proxy" containerID="cri-o://f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b" gracePeriod=30
Jul 28 11:02:10.517016 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:10.516982    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c3b9a5_3fd5_4554_a3f0_693649c4efb6.slice/crio-conmon-f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 11:02:11.348703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:11.348654    2569 generic.go:356] "Generic (PLEG): container finished" podID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerID="f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b" exitCode=2
Jul 28 11:02:11.348876 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:11.348720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerDied","Data":"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"}
Jul 28 11:02:12.013743 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:12.013676    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 28 11:02:12.018311 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:12.018282    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 11:02:14.124956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.124929    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:02:14.246022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.245986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls\") pod \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") "
Jul 28 11:02:14.246022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246030    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9w7n\" (UniqueName: \"kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n\") pod \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") "
Jul 28 11:02:14.246295 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246056    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config\") pod \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") "
Jul 28 11:02:14.246295 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location\") pod \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") "
Jul 28 11:02:14.246295 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246187    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert\") pod \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\" (UID: \"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6\") "
Jul 28 11:02:14.246495 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246465    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" (UID: "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:02:14.246559 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246466    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" (UID: "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:02:14.246559 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.246528    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" (UID: "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:02:14.248219 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.248195    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n" (OuterVolumeSpecName: "kube-api-access-t9w7n") pod "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" (UID: "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6"). InnerVolumeSpecName "kube-api-access-t9w7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:02:14.248310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.248219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" (UID: "f9c3b9a5-3fd5-4554-a3f0-693649c4efb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:02:14.347161 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.347126    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:02:14.347161 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.347154    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:02:14.347161 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.347167    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:02:14.347381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.347176    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:02:14.347381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.347184    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9w7n\" (UniqueName: \"kubernetes.io/projected/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6-kube-api-access-t9w7n\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:02:14.360972 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.360945    2569 generic.go:356] "Generic (PLEG): container finished" podID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerID="400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c" exitCode=0
Jul 28 11:02:14.361106 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.361000    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerDied","Data":"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"}
Jul 28 11:02:14.361106 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.361060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66" event={"ID":"f9c3b9a5-3fd5-4554-a3f0-693649c4efb6","Type":"ContainerDied","Data":"c08cdb5956bcb5d46b7b91ab008a12c0de42cdbe1f0ee5021d0751f024975a0d"}
Jul 28 11:02:14.361106 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.361086    2569 scope.go:117] "RemoveContainer" containerID="f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"
Jul 28 11:02:14.361210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.361011    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"
Jul 28 11:02:14.369211 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.369190    2569 scope.go:117] "RemoveContainer" containerID="400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"
Jul 28 11:02:14.376418 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.376401    2569 scope.go:117] "RemoveContainer" containerID="e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"
Jul 28 11:02:14.382911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.382887    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:02:14.383448 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.383434    2569 scope.go:117] "RemoveContainer" containerID="f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"
Jul 28 11:02:14.383719 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:14.383665    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b\": container with ID starting with f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b not found: ID does not exist" containerID="f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"
Jul 28 11:02:14.383810 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.383718    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b"} err="failed to get container status \"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b\": rpc error: code = NotFound desc = could not find container \"f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b\": container with ID starting with f661500efa8ee7afbea7ddc38bd286ce4e16cee132c31678be7535f352cd6a3b not found: ID does not exist"
Jul 28 11:02:14.383810 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.383744    2569 scope.go:117] "RemoveContainer" containerID="400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"
Jul 28 11:02:14.383978 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:14.383961    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c\": container with ID starting with 400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c not found: ID does not exist" containerID="400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"
Jul 28 11:02:14.384040 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.383984    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c"} err="failed to get container status \"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c\": rpc error: code = NotFound desc = could not find container \"400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c\": container with ID starting with 400e069eaebf21888e3555b975998711a7690d361b76f40db42392c50f6c406c not found: ID does not exist"
Jul 28 11:02:14.384040 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.384004    2569 scope.go:117] "RemoveContainer" containerID="e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"
Jul 28 11:02:14.384211 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:14.384196    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7\": container with ID starting with e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7 not found: ID does not exist" containerID="e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"
Jul 28 11:02:14.384262 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.384216    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7"} err="failed to get container status \"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7\": rpc error: code = NotFound desc = could not find container \"e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7\": container with ID starting with e9e5b02a64e305b5775517f099edb099e80bba6f5351d06ff4df16fa4c6ce0f7 not found: ID does not exist"
Jul 28 11:02:14.388277 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.388255    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5p66"]
Jul 28 11:02:14.566633 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:14.566548    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" path="/var/lib/kubelet/pods/f9c3b9a5-3fd5-4554-a3f0-693649c4efb6/volumes"
Jul 28 11:02:15.642709 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.642647    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643099    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="storage-initializer"
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643117    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="storage-initializer"
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643128    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container"
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643138    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container"
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643165    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kube-rbac-proxy"
Jul 28 11:02:15.643202 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643173    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kube-rbac-proxy"
Jul 28 11:02:15.643533 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643247    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kube-rbac-proxy"
Jul 28 11:02:15.643533 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.643257    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c3b9a5-3fd5-4554-a3f0-693649c4efb6" containerName="kserve-container"
Jul 28 11:02:15.648102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.648078    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.651287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-13740c-predictor-serving-cert\""
Jul 28 11:02:15.651405 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651299    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:02:15.651405 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651308    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:02:15.651405 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651338    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:02:15.651571 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651475    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:02:15.651622 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.651579    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:02:15.652093 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.652076    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-13740c-kube-rbac-proxy-sar-config\""
Jul 28 11:02:15.658483 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.658462    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:02:15.760189 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.760157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjrw\" (UniqueName: \"kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.760189 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.760193    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.760397 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.760210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.760397 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.760313    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.760397 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.760353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.861510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.861468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjrw\" (UniqueName: \"kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.861510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.861515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.861795 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.861539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.861795 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:15.861669    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-13740c-predictor-serving-cert: secret "isvc-primary-13740c-predictor-serving-cert" not found
Jul 28 11:02:15.861795 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.861714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.861795 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:02:15.861751    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls podName:5da45863-f84c-4fd5-a784-31126d8d667e nodeName:}" failed. No retries permitted until 2026-07-28 11:02:16.361730965 +0000 UTC m=+3294.282435004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls") pod "isvc-primary-13740c-predictor-86b67994b4-875jt" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e") : secret "isvc-primary-13740c-predictor-serving-cert" not found
Jul 28 11:02:15.862039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.861794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.862137 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.862114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.862370 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.862349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.862439 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.862383    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:15.872418 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:15.872394    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjrw\" (UniqueName: \"kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:16.366438 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:16.366403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:16.369048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:16.369019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") pod \"isvc-primary-13740c-predictor-86b67994b4-875jt\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") " pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:16.558951 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:16.558902    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:16.688201 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:16.688174    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:02:16.690501 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:02:16.690470    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da45863_f84c_4fd5_a784_31126d8d667e.slice/crio-a2edd0a8d651d303a060f17501879397d563727bf0cf8039cdd6b7570f15f549 WatchSource:0}: Error finding container a2edd0a8d651d303a060f17501879397d563727bf0cf8039cdd6b7570f15f549: Status 404 returned error can't find the container with id a2edd0a8d651d303a060f17501879397d563727bf0cf8039cdd6b7570f15f549
Jul 28 11:02:17.374936 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:17.374901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerStarted","Data":"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"}
Jul 28 11:02:17.375122 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:17.374963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerStarted","Data":"a2edd0a8d651d303a060f17501879397d563727bf0cf8039cdd6b7570f15f549"}
Jul 28 11:02:21.392382 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:21.392347    2569 generic.go:356] "Generic (PLEG): container finished" podID="5da45863-f84c-4fd5-a784-31126d8d667e" containerID="67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07" exitCode=0
Jul 28 11:02:21.392808 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:21.392391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerDied","Data":"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"}
Jul 28 11:02:22.399843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.399801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerStarted","Data":"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"}
Jul 28 11:02:22.400322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.399873    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerStarted","Data":"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"}
Jul 28 11:02:22.400322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.400175    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:22.419591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.419539    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podStartSLOduration=3.746601459 podStartE2EDuration="7.419522384s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672920925s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:22.418347466 +0000 UTC m=+3300.339051525" watchObservedRunningTime="2026-07-28 11:02:22.419522384 +0000 UTC m=+3300.340226435"
Jul 28 11:02:22.667589 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.667507    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:02:22.677606 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:22.677586    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:02:23.404189 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:23.404157    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:23.405321 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:23.405298    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:02:24.408153 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:24.408120    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:02:29.412465 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:29.412433    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:02:29.412897 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:29.412871    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:02:39.413046 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:39.413007    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:02:49.413877 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:49.413790    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:02:59.413551 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:02:59.413508    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:03:09.413225 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:03:09.413187    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:03:19.413314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:03:19.413272    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:03:29.412905 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:03:29.412866    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:03:39.413434 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:03:39.413404    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:04:00.861459 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.861425    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:00.864844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.864829    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:00.867591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.867568    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-13740c\""
Jul 28 11:04:00.867591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.867582    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-13740c-kube-rbac-proxy-sar-config\""
Jul 28 11:04:00.867797 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.867663    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-13740c-predictor-serving-cert\""
Jul 28 11:04:00.867920 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.867906    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-13740c-dockercfg-t4xz8\""
Jul 28 11:04:00.874713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.874675    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:00.929609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.929582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:00.929788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.929614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:00.929788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.929658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:00.929788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.929676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:00.929788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:00.929723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9n8\" (UniqueName: \"kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.030847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.030810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.030851    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.030896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9n8\" (UniqueName: \"kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.030955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031246 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.031175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031484 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.031464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031575 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.031518    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.031635 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.031584    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.033383 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.033363    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.038964 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.038942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9n8\" (UniqueName: \"kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8\") pod \"isvc-secondary-13740c-predictor-bdc767467-88gcr\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") " pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.175968 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.175931    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:01.303535 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.303434    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:01.306166 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:04:01.306137    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7ebe59_afb3_4267_b0af_2899b324e1a9.slice/crio-b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f WatchSource:0}: Error finding container b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f: Status 404 returned error can't find the container with id b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f
Jul 28 11:04:01.793554 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.793513    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerStarted","Data":"9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"}
Jul 28 11:04:01.793754 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:01.793603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerStarted","Data":"b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f"}
Jul 28 11:04:06.815555 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:06.815529    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/0.log"
Jul 28 11:04:06.815993 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:06.815566    2569 generic.go:356] "Generic (PLEG): container finished" podID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerID="9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5" exitCode=1
Jul 28 11:04:06.815993 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:06.815607    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerDied","Data":"9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"}
Jul 28 11:04:07.821320 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:07.821292    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/0.log"
Jul 28 11:04:07.821720 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:07.821387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerStarted","Data":"a630d96e4529d525d50be1c9a60692d732599374dee803e2f101ed1003a2e0bf"}
Jul 28 11:04:10.834377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.834345    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/1.log"
Jul 28 11:04:10.834843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.834721    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/0.log"
Jul 28 11:04:10.834843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.834753    2569 generic.go:356] "Generic (PLEG): container finished" podID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerID="a630d96e4529d525d50be1c9a60692d732599374dee803e2f101ed1003a2e0bf" exitCode=1
Jul 28 11:04:10.834843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.834832    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerDied","Data":"a630d96e4529d525d50be1c9a60692d732599374dee803e2f101ed1003a2e0bf"}
Jul 28 11:04:10.834997 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.834897    2569 scope.go:117] "RemoveContainer" containerID="9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"
Jul 28 11:04:10.835318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.835300    2569 scope.go:117] "RemoveContainer" containerID="9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"
Jul 28 11:04:10.845874 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:10.845846    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-13740c-predictor-bdc767467-88gcr_kserve-ci-e2e-test(4b7ebe59-afb3-4267-b0af-2899b324e1a9)\"" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9"
Jul 28 11:04:10.846164 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:10.846137    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-13740c-predictor-bdc767467-88gcr_kserve-ci-e2e-test_4b7ebe59-afb3-4267-b0af-2899b324e1a9_0 in pod sandbox b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f from index: no such id: '9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5'" containerID="9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"
Jul 28 11:04:10.846241 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:10.846173    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-13740c-predictor-bdc767467-88gcr_kserve-ci-e2e-test_4b7ebe59-afb3-4267-b0af-2899b324e1a9_0 in pod sandbox b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f from index: no such id: '9c5526c86ee571ec261263f4fb133d00f926b2dbdc40c3364be1254fe4196bc5'"
Jul 28 11:04:11.840040 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:11.840010    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/1.log"
Jul 28 11:04:18.916555 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:18.916475    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:18.974496 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:18.974459    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:04:18.974820 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:18.974794    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" containerID="cri-o://66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500" gracePeriod=30
Jul 28 11:04:18.974934 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:18.974884    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kube-rbac-proxy" containerID="cri-o://3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa" gracePeriod=30
Jul 28 11:04:19.095111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.095089    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/1.log"
Jul 28 11:04:19.097429 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.097404    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:19.187583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls\") pod \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") "
Jul 28 11:04:19.187583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187547    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location\") pod \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") "
Jul 28 11:04:19.187849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187607    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config\") pod \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") "
Jul 28 11:04:19.187849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187657    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9n8\" (UniqueName: \"kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8\") pod \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") "
Jul 28 11:04:19.187849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187740    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert\") pod \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\" (UID: \"4b7ebe59-afb3-4267-b0af-2899b324e1a9\") "
Jul 28 11:04:19.188016 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.187909    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b7ebe59-afb3-4267-b0af-2899b324e1a9" (UID: "4b7ebe59-afb3-4267-b0af-2899b324e1a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:19.188073 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.188029    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:19.188115 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.188059    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b7ebe59-afb3-4267-b0af-2899b324e1a9" (UID: "4b7ebe59-afb3-4267-b0af-2899b324e1a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:19.188156 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.188141    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b7ebe59-afb3-4267-b0af-2899b324e1a9" (UID: "4b7ebe59-afb3-4267-b0af-2899b324e1a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:19.189731 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.189657    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b7ebe59-afb3-4267-b0af-2899b324e1a9" (UID: "4b7ebe59-afb3-4267-b0af-2899b324e1a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:04:19.189839 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.189774    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8" (OuterVolumeSpecName: "kube-api-access-dm9n8") pod "4b7ebe59-afb3-4267-b0af-2899b324e1a9" (UID: "4b7ebe59-afb3-4267-b0af-2899b324e1a9"). InnerVolumeSpecName "kube-api-access-dm9n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:04:19.289330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.289292    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:19.289330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.289323    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9n8\" (UniqueName: \"kubernetes.io/projected/4b7ebe59-afb3-4267-b0af-2899b324e1a9-kube-api-access-dm9n8\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:19.289330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.289333    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b7ebe59-afb3-4267-b0af-2899b324e1a9-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:19.289330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.289343    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b7ebe59-afb3-4267-b0af-2899b324e1a9-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:19.408849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.408807    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 28 11:04:19.413014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.412990    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 11:04:19.875392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.875359    2569 generic.go:356] "Generic (PLEG): container finished" podID="5da45863-f84c-4fd5-a784-31126d8d667e" containerID="3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa" exitCode=2
Jul 28 11:04:19.875585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.875429    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerDied","Data":"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"}
Jul 28 11:04:19.876557 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.876543    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-13740c-predictor-bdc767467-88gcr_4b7ebe59-afb3-4267-b0af-2899b324e1a9/storage-initializer/1.log"
Jul 28 11:04:19.876625 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.876601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr" event={"ID":"4b7ebe59-afb3-4267-b0af-2899b324e1a9","Type":"ContainerDied","Data":"b72f1e13f83525075444ad44c1554004284661fe30da2223441f04868d8ecb6f"}
Jul 28 11:04:19.876670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.876633    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"
Jul 28 11:04:19.876670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.876661    2569 scope.go:117] "RemoveContainer" containerID="a630d96e4529d525d50be1c9a60692d732599374dee803e2f101ed1003a2e0bf"
Jul 28 11:04:19.912659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.912631    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:19.916620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:19.916596    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-13740c-predictor-bdc767467-88gcr"]
Jul 28 11:04:20.566171 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:20.566132    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" path="/var/lib/kubelet/pods/4b7ebe59-afb3-4267-b0af-2899b324e1a9/volumes"
Jul 28 11:04:23.719933 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.719909    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:04:23.831234 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831155    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert\") pod \"5da45863-f84c-4fd5-a784-31126d8d667e\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") "
Jul 28 11:04:23.831234 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831209    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config\") pod \"5da45863-f84c-4fd5-a784-31126d8d667e\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") "
Jul 28 11:04:23.831234 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location\") pod \"5da45863-f84c-4fd5-a784-31126d8d667e\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") "
Jul 28 11:04:23.831472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831256    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") pod \"5da45863-f84c-4fd5-a784-31126d8d667e\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") "
Jul 28 11:04:23.831472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kjrw\" (UniqueName: \"kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw\") pod \"5da45863-f84c-4fd5-a784-31126d8d667e\" (UID: \"5da45863-f84c-4fd5-a784-31126d8d667e\") "
Jul 28 11:04:23.831574 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5da45863-f84c-4fd5-a784-31126d8d667e" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:23.831619 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831558    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5da45863-f84c-4fd5-a784-31126d8d667e" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:23.831619 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.831571    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5da45863-f84c-4fd5-a784-31126d8d667e" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:23.833354 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.833333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5da45863-f84c-4fd5-a784-31126d8d667e" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:04:23.833424 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.833378    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw" (OuterVolumeSpecName: "kube-api-access-9kjrw") pod "5da45863-f84c-4fd5-a784-31126d8d667e" (UID: "5da45863-f84c-4fd5-a784-31126d8d667e"). InnerVolumeSpecName "kube-api-access-9kjrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:04:23.895047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.895009    2569 generic.go:356] "Generic (PLEG): container finished" podID="5da45863-f84c-4fd5-a784-31126d8d667e" containerID="66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500" exitCode=0
Jul 28 11:04:23.895239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.895071    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"
Jul 28 11:04:23.895239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.895093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerDied","Data":"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"}
Jul 28 11:04:23.895239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.895154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt" event={"ID":"5da45863-f84c-4fd5-a784-31126d8d667e","Type":"ContainerDied","Data":"a2edd0a8d651d303a060f17501879397d563727bf0cf8039cdd6b7570f15f549"}
Jul 28 11:04:23.895239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.895184    2569 scope.go:117] "RemoveContainer" containerID="3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"
Jul 28 11:04:23.903304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.903205    2569 scope.go:117] "RemoveContainer" containerID="66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"
Jul 28 11:04:23.910466 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.910448    2569 scope.go:117] "RemoveContainer" containerID="67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"
Jul 28 11:04:23.917440 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.917418    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:04:23.917930 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.917913    2569 scope.go:117] "RemoveContainer" containerID="3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"
Jul 28 11:04:23.918178 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:23.918159    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa\": container with ID starting with 3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa not found: ID does not exist" containerID="3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"
Jul 28 11:04:23.918228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.918187    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa"} err="failed to get container status \"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa\": rpc error: code = NotFound desc = could not find container \"3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa\": container with ID starting with 3a1bc0de3a595cb3373b4c85bae7b0bfecc2ee0109dec977e396a2ca3051aefa not found: ID does not exist"
Jul 28 11:04:23.918228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.918206    2569 scope.go:117] "RemoveContainer" containerID="66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"
Jul 28 11:04:23.918446 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:23.918428    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500\": container with ID starting with 66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500 not found: ID does not exist" containerID="66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"
Jul 28 11:04:23.918490 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.918452    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500"} err="failed to get container status \"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500\": rpc error: code = NotFound desc = could not find container \"66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500\": container with ID starting with 66b57f5f57c846047bbcd13cf98c9163d7c50b64f6779e00eb7b3e8cd24bb500 not found: ID does not exist"
Jul 28 11:04:23.918490 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.918468    2569 scope.go:117] "RemoveContainer" containerID="67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"
Jul 28 11:04:23.918704 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:23.918672    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07\": container with ID starting with 67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07 not found: ID does not exist" containerID="67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"
Jul 28 11:04:23.918754 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.918700    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07"} err="failed to get container status \"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07\": rpc error: code = NotFound desc = could not find container \"67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07\": container with ID starting with 67498e0c5f33b436316df396eff1d5183c11ffe5fa0c87fe85db93094b934d07 not found: ID does not exist"
Jul 28 11:04:23.921173 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.921155    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-13740c-predictor-86b67994b4-875jt"]
Jul 28 11:04:23.932856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.932836    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:23.932856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.932855    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5da45863-f84c-4fd5-a784-31126d8d667e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:23.932974 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.932866    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5da45863-f84c-4fd5-a784-31126d8d667e-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:23.932974 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.932876    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5da45863-f84c-4fd5-a784-31126d8d667e-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:23.932974 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:23.932884    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjrw\" (UniqueName: \"kubernetes.io/projected/5da45863-f84c-4fd5-a784-31126d8d667e-kube-api-access-9kjrw\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:24.110571 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110494    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:24.110834 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110821    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110837    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110848    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kube-rbac-proxy"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110854    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kube-rbac-proxy"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110861    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="storage-initializer"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110867    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="storage-initializer"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110879    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.110884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110884    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110898    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110904    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110950    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kube-rbac-proxy"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110958    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110964    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7ebe59-afb3-4267-b0af-2899b324e1a9" containerName="storage-initializer"
Jul 28 11:04:24.111102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.110971    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" containerName="kserve-container"
Jul 28 11:04:24.115637 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.115620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.118204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118181    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-c75926-dockercfg-4fkh4\""
Jul 28 11:04:24.118357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118181    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:04:24.118357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118216    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c75926-kube-rbac-proxy-sar-config\""
Jul 28 11:04:24.118357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118252    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:04:24.118493 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118409    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c75926-predictor-serving-cert\""
Jul 28 11:04:24.118596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118580    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-c75926\""
Jul 28 11:04:24.118651 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.118599    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:04:24.123172 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.123153    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:24.234968 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.234931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.235166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.234976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbd4\" (UniqueName: \"kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.235166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.235001    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.235166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.235068    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.235166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.235130    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336121 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336085    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336257 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336324 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336468 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbd4\" (UniqueName: \"kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336543 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.336983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.336962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.338742 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.338723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.344430 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.344407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbd4\" (UniqueName: \"kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4\") pod \"isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") " pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.426915 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.426884    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:24.555672 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.555494    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:24.558337 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:04:24.558309    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45870883_4da8_42ba_b9d0_70d696f27a13.slice/crio-13ba70aa7cb742ce4e75d3befa4aa4e1d5bb4692052f033eb7e7e02398981381 WatchSource:0}: Error finding container 13ba70aa7cb742ce4e75d3befa4aa4e1d5bb4692052f033eb7e7e02398981381: Status 404 returned error can't find the container with id 13ba70aa7cb742ce4e75d3befa4aa4e1d5bb4692052f033eb7e7e02398981381
Jul 28 11:04:24.565918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.565895    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da45863-f84c-4fd5-a784-31126d8d667e" path="/var/lib/kubelet/pods/5da45863-f84c-4fd5-a784-31126d8d667e/volumes"
Jul 28 11:04:24.901427 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.901392    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerStarted","Data":"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"}
Jul 28 11:04:24.901859 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:24.901463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerStarted","Data":"13ba70aa7cb742ce4e75d3befa4aa4e1d5bb4692052f033eb7e7e02398981381"}
Jul 28 11:04:26.912336 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:26.912309    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/0.log"
Jul 28 11:04:26.912743 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:26.912346    2569 generic.go:356] "Generic (PLEG): container finished" podID="45870883-4da8-42ba-b9d0-70d696f27a13" containerID="bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73" exitCode=1
Jul 28 11:04:26.912743 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:26.912391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerDied","Data":"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"}
Jul 28 11:04:27.918135 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:27.918106    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/0.log"
Jul 28 11:04:27.918497 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:27.918199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerStarted","Data":"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"}
Jul 28 11:04:29.145616 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.145584    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:29.146127 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.145831    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer" containerID="cri-o://1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232" gracePeriod=30
Jul 28 11:04:29.887512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.887489    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/1.log"
Jul 28 11:04:29.887907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.887891    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/0.log"
Jul 28 11:04:29.890173 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.890159    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:29.927763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.927736    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/1.log"
Jul 28 11:04:29.928109 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928096    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4_45870883-4da8-42ba-b9d0-70d696f27a13/storage-initializer/0.log"
Jul 28 11:04:29.928164 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928127    2569 generic.go:356] "Generic (PLEG): container finished" podID="45870883-4da8-42ba-b9d0-70d696f27a13" containerID="1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232" exitCode=1
Jul 28 11:04:29.928206 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerDied","Data":"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"}
Jul 28 11:04:29.928206 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928180    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"
Jul 28 11:04:29.928272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928221    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4" event={"ID":"45870883-4da8-42ba-b9d0-70d696f27a13","Type":"ContainerDied","Data":"13ba70aa7cb742ce4e75d3befa4aa4e1d5bb4692052f033eb7e7e02398981381"}
Jul 28 11:04:29.928272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.928250    2569 scope.go:117] "RemoveContainer" containerID="1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"
Jul 28 11:04:29.936191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.936178    2569 scope.go:117] "RemoveContainer" containerID="bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"
Jul 28 11:04:29.943061 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.943046    2569 scope.go:117] "RemoveContainer" containerID="1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"
Jul 28 11:04:29.943300 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:29.943282    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232\": container with ID starting with 1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232 not found: ID does not exist" containerID="1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"
Jul 28 11:04:29.943348 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.943313    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232"} err="failed to get container status \"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232\": rpc error: code = NotFound desc = could not find container \"1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232\": container with ID starting with 1e144bcb262aebf56f2c86aa4dd8f0eb6fc132254a3fb3bd345db0beb1fff232 not found: ID does not exist"
Jul 28 11:04:29.943348 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.943330    2569 scope.go:117] "RemoveContainer" containerID="bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"
Jul 28 11:04:29.943532 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:04:29.943518    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73\": container with ID starting with bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73 not found: ID does not exist" containerID="bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"
Jul 28 11:04:29.943586 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.943537    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73"} err="failed to get container status \"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73\": rpc error: code = NotFound desc = could not find container \"bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73\": container with ID starting with bb775e04710b865f18885e587e2e0864f397a2263fa8c412a7875a9bc5a0ed73 not found: ID does not exist"
Jul 28 11:04:29.984938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.984858    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert\") pod \"45870883-4da8-42ba-b9d0-70d696f27a13\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") "
Jul 28 11:04:29.984938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.984902    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctbd4\" (UniqueName: \"kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4\") pod \"45870883-4da8-42ba-b9d0-70d696f27a13\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") "
Jul 28 11:04:29.985124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.984950    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls\") pod \"45870883-4da8-42ba-b9d0-70d696f27a13\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") "
Jul 28 11:04:29.985124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.984986    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config\") pod \"45870883-4da8-42ba-b9d0-70d696f27a13\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") "
Jul 28 11:04:29.985124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.985007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location\") pod \"45870883-4da8-42ba-b9d0-70d696f27a13\" (UID: \"45870883-4da8-42ba-b9d0-70d696f27a13\") "
Jul 28 11:04:29.985353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.985262    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45870883-4da8-42ba-b9d0-70d696f27a13" (UID: "45870883-4da8-42ba-b9d0-70d696f27a13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:29.985409 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.985337    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45870883-4da8-42ba-b9d0-70d696f27a13" (UID: "45870883-4da8-42ba-b9d0-70d696f27a13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:29.985409 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.985368    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45870883-4da8-42ba-b9d0-70d696f27a13" (UID: "45870883-4da8-42ba-b9d0-70d696f27a13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:29.987084 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.987047    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45870883-4da8-42ba-b9d0-70d696f27a13" (UID: "45870883-4da8-42ba-b9d0-70d696f27a13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:04:29.987084 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:29.987060    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4" (OuterVolumeSpecName: "kube-api-access-ctbd4") pod "45870883-4da8-42ba-b9d0-70d696f27a13" (UID: "45870883-4da8-42ba-b9d0-70d696f27a13"). InnerVolumeSpecName "kube-api-access-ctbd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:04:30.085587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.085541    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45870883-4da8-42ba-b9d0-70d696f27a13-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:30.085587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.085582    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:30.085587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.085595    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbd4\" (UniqueName: \"kubernetes.io/projected/45870883-4da8-42ba-b9d0-70d696f27a13-kube-api-access-ctbd4\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:30.085587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.085604    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45870883-4da8-42ba-b9d0-70d696f27a13-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:30.085887 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.085612    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45870883-4da8-42ba-b9d0-70d696f27a13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:04:30.264275 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.264246    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:30.268316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.268293    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c75926-predictor-57b854d7bf-6bgd4"]
Jul 28 11:04:30.566375 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:30.566295    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" path="/var/lib/kubelet/pods/45870883-4da8-42ba-b9d0-70d696f27a13/volumes"
Jul 28 11:04:34.344038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.343984    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344304    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344315    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344323    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344329    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344387    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.344423 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.344394    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="45870883-4da8-42ba-b9d0-70d696f27a13" containerName="storage-initializer"
Jul 28 11:04:34.348801 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.348784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.351374 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351349    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:04:34.351491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351377    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 11:04:34.351491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351395    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:04:34.351491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351355    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 11:04:34.351491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351355    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:04:34.351491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.351462    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:04:34.352282 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.352270    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:04:34.357272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.357251    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:04:34.523746 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.523710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.523923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.523758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.523923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.523822    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.523923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.523868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.524029 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.523926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkx8\" (UniqueName: \"kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.624623 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.624543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkx8\" (UniqueName: \"kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.624623 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.624593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.624623 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.624621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.624945 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.624646    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.624945 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.624680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.625064 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.625044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.625298 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.625270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.625397 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.625347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.627132 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.627110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.632652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.632633    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkx8\" (UniqueName: \"kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-lnmbg\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.659373 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.659350    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:04:34.792770 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.792746    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:04:34.795293 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:04:34.795267    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffbb120_c661_4cce_8737_01ddc2bb0260.slice/crio-b77fc72575cff5851cd981a09cb7d32e7f07bc93d745f3732cc7d49e046cc6bc WatchSource:0}: Error finding container b77fc72575cff5851cd981a09cb7d32e7f07bc93d745f3732cc7d49e046cc6bc: Status 404 returned error can't find the container with id b77fc72575cff5851cd981a09cb7d32e7f07bc93d745f3732cc7d49e046cc6bc
Jul 28 11:04:34.948848 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.948811    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerStarted","Data":"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"}
Jul 28 11:04:34.949031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:34.948885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerStarted","Data":"b77fc72575cff5851cd981a09cb7d32e7f07bc93d745f3732cc7d49e046cc6bc"}
Jul 28 11:04:38.967068 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:38.967032    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerID="208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c" exitCode=0
Jul 28 11:04:38.967505 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:38.967102    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerDied","Data":"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"}
Jul 28 11:04:38.968263 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:04:38.968250    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:05:02.086479 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:02.086442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerStarted","Data":"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"}
Jul 28 11:05:02.087035 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:02.086544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerStarted","Data":"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"}
Jul 28 11:05:02.087035 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:02.086790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:05:02.106636 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:02.106579    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podStartSLOduration=2.320088448 podStartE2EDuration="28.106564806s" totalImagesPullingTime="22.124017127s" totalInitContainerRuntime="3.662459231s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:05:02.105886643 +0000 UTC m=+3460.026590696" watchObservedRunningTime="2026-07-28 11:05:02.106564806 +0000 UTC m=+3460.027268858"
Jul 28 11:05:03.091583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:03.091544    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:05:03.092833 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:03.092804    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:04.095696 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:04.095644    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:09.100198 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:09.100163    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:05:09.100759 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:09.100727    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:19.100592 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:19.100552    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:29.101514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:29.101473    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:39.101065 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:39.101020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:49.101427 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:49.101338    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:05:59.101292 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:05:59.101254    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:06:09.101605 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:09.101566    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:06:19.102145 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:19.102106    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:06:24.490901 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:24.490865    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:06:24.491340 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:24.491182    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" containerID="cri-o://78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f" gracePeriod=30
Jul 28 11:06:24.491340 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:24.491207    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kube-rbac-proxy" containerID="cri-o://ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517" gracePeriod=30
Jul 28 11:06:25.429936 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:25.429902    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerID="ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517" exitCode=2
Jul 28 11:06:25.430115 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:25.429967    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerDied","Data":"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"}
Jul 28 11:06:29.096474 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.096429    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 28 11:06:29.100828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.100797    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 11:06:29.240975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.240950    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:06:29.280669 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.280634    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls\") pod \"6ffbb120-c661-4cce-8737-01ddc2bb0260\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") "
Jul 28 11:06:29.280882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.280708    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config\") pod \"6ffbb120-c661-4cce-8737-01ddc2bb0260\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") "
Jul 28 11:06:29.280882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.280743    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqkx8\" (UniqueName: \"kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8\") pod \"6ffbb120-c661-4cce-8737-01ddc2bb0260\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") "
Jul 28 11:06:29.280882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.280772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location\") pod \"6ffbb120-c661-4cce-8737-01ddc2bb0260\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") "
Jul 28 11:06:29.280882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.280803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert\") pod \"6ffbb120-c661-4cce-8737-01ddc2bb0260\" (UID: \"6ffbb120-c661-4cce-8737-01ddc2bb0260\") "
Jul 28 11:06:29.281135 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.281096    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ffbb120-c661-4cce-8737-01ddc2bb0260" (UID: "6ffbb120-c661-4cce-8737-01ddc2bb0260"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:06:29.281135 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.281100    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ffbb120-c661-4cce-8737-01ddc2bb0260" (UID: "6ffbb120-c661-4cce-8737-01ddc2bb0260"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:06:29.281304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.281205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ffbb120-c661-4cce-8737-01ddc2bb0260" (UID: "6ffbb120-c661-4cce-8737-01ddc2bb0260"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:06:29.282844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.282817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ffbb120-c661-4cce-8737-01ddc2bb0260" (UID: "6ffbb120-c661-4cce-8737-01ddc2bb0260"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:06:29.282844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.282823    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8" (OuterVolumeSpecName: "kube-api-access-qqkx8") pod "6ffbb120-c661-4cce-8737-01ddc2bb0260" (UID: "6ffbb120-c661-4cce-8737-01ddc2bb0260"). InnerVolumeSpecName "kube-api-access-qqkx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:06:29.381578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.381491    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ffbb120-c661-4cce-8737-01ddc2bb0260-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:06:29.381578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.381521    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:06:29.381578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.381532    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqkx8\" (UniqueName: \"kubernetes.io/projected/6ffbb120-c661-4cce-8737-01ddc2bb0260-kube-api-access-qqkx8\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:06:29.381578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.381541    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ffbb120-c661-4cce-8737-01ddc2bb0260-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:06:29.381578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.381550    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ffbb120-c661-4cce-8737-01ddc2bb0260-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:06:29.447079 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.447043    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerID="78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f" exitCode=0
Jul 28 11:06:29.447237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.447099    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"
Jul 28 11:06:29.447237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.447132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerDied","Data":"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"}
Jul 28 11:06:29.447237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.447190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg" event={"ID":"6ffbb120-c661-4cce-8737-01ddc2bb0260","Type":"ContainerDied","Data":"b77fc72575cff5851cd981a09cb7d32e7f07bc93d745f3732cc7d49e046cc6bc"}
Jul 28 11:06:29.447237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.447218    2569 scope.go:117] "RemoveContainer" containerID="ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"
Jul 28 11:06:29.455458 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.455438    2569 scope.go:117] "RemoveContainer" containerID="78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"
Jul 28 11:06:29.462673 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.462651    2569 scope.go:117] "RemoveContainer" containerID="208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"
Jul 28 11:06:29.469934 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.469910    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:06:29.471003 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.470974    2569 scope.go:117] "RemoveContainer" containerID="ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"
Jul 28 11:06:29.471285 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:06:29.471265    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517\": container with ID starting with ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517 not found: ID does not exist" containerID="ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"
Jul 28 11:06:29.471344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.471295    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517"} err="failed to get container status \"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517\": rpc error: code = NotFound desc = could not find container \"ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517\": container with ID starting with ee57d242071dc438260d47dca8f10fa743ec7124d01c1643ba36033aa4317517 not found: ID does not exist"
Jul 28 11:06:29.471344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.471314    2569 scope.go:117] "RemoveContainer" containerID="78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"
Jul 28 11:06:29.471521 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:06:29.471503    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f\": container with ID starting with 78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f not found: ID does not exist" containerID="78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"
Jul 28 11:06:29.471566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.471528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f"} err="failed to get container status \"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f\": rpc error: code = NotFound desc = could not find container \"78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f\": container with ID starting with 78b57ebfa0fdb1ba8a4f99599224e6a06ff0c61f83b976fa9fa6013ed7ea804f not found: ID does not exist"
Jul 28 11:06:29.471566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.471544    2569 scope.go:117] "RemoveContainer" containerID="208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"
Jul 28 11:06:29.471774 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:06:29.471756    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c\": container with ID starting with 208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c not found: ID does not exist" containerID="208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"
Jul 28 11:06:29.471851 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.471782    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c"} err="failed to get container status \"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c\": rpc error: code = NotFound desc = could not find container \"208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c\": container with ID starting with 208fff32e07607d24a3d6735f1f6aa7e4711a2eb51e86051047146b10363707c not found: ID does not exist"
Jul 28 11:06:29.475241 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.475217    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-lnmbg"]
Jul 28 11:06:29.654462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:06:29.654839 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="storage-initializer"
Jul 28 11:06:29.654941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654841    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="storage-initializer"
Jul 28 11:06:29.654941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654860    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kube-rbac-proxy"
Jul 28 11:06:29.654941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654868    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kube-rbac-proxy"
Jul 28 11:06:29.654941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654896    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container"
Jul 28 11:06:29.654941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654904    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container"
Jul 28 11:06:29.655213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654980    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kserve-container"
Jul 28 11:06:29.655213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.654994    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" containerName="kube-rbac-proxy"
Jul 28 11:06:29.658004 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.657984    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.660656 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660621    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:06:29.660656 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660647    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 11:06:29.660844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660676    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:06:29.660844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660720    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:06:29.660844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660674    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 11:06:29.660844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660648    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:06:29.660844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.660678    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:06:29.669093 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.669070    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:06:29.682996 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.682973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.683116 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.683021    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.683116 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.683073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89shv\" (UniqueName: \"kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.683220 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.683122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.683220 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.683186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784218 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784419 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784419 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89shv\" (UniqueName: \"kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784419 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784368    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.784835 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.784814    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.785125 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.785103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.785169 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.785103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.786758 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.786737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.796039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.796019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89shv\" (UniqueName: \"kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv\") pod \"isvc-predictive-xgboost-predictor-695f978978-7b2nx\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:29.968144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:29.968050    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:30.095357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:30.095321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:06:30.098194 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:06:30.098164    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074dd06a_204f_48fd_9d11_7f8f7223a226.slice/crio-2431e121842d2a5e05e7e54ab7a268f1b1e99d3a6b2b1fda1640b579c1d344c4 WatchSource:0}: Error finding container 2431e121842d2a5e05e7e54ab7a268f1b1e99d3a6b2b1fda1640b579c1d344c4: Status 404 returned error can't find the container with id 2431e121842d2a5e05e7e54ab7a268f1b1e99d3a6b2b1fda1640b579c1d344c4
Jul 28 11:06:30.452274 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:30.452233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerStarted","Data":"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"}
Jul 28 11:06:30.452467 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:30.452308    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerStarted","Data":"2431e121842d2a5e05e7e54ab7a268f1b1e99d3a6b2b1fda1640b579c1d344c4"}
Jul 28 11:06:30.567236 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:30.567191    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ffbb120-c661-4cce-8737-01ddc2bb0260" path="/var/lib/kubelet/pods/6ffbb120-c661-4cce-8737-01ddc2bb0260/volumes"
Jul 28 11:06:34.470955 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:34.470920    2569 generic.go:356] "Generic (PLEG): container finished" podID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerID="19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a" exitCode=0
Jul 28 11:06:34.471350 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:34.470989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerDied","Data":"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"}
Jul 28 11:06:35.480983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.480942    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerStarted","Data":"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"}
Jul 28 11:06:35.481389 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.481008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerStarted","Data":"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"}
Jul 28 11:06:35.481389 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.481262    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:35.481465 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.481394    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:35.482610 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.482587    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:06:35.501356 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:35.501306    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podStartSLOduration=2.791811686 podStartE2EDuration="6.50129152s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.709479834s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:35.499381231 +0000 UTC m=+3553.420085282" watchObservedRunningTime="2026-07-28 11:06:35.50129152 +0000 UTC m=+3553.421995573"
Jul 28 11:06:36.485226 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:36.485185    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:06:41.489658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:41.489620    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:06:41.490219 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:41.490196    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:06:51.490961 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:06:51.490919    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:01.490255 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:01.490211    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:11.490615 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:11.490578    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:21.490456 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:21.490417    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:22.694977 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:22.694951    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:07:22.704504 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:22.704480    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:07:31.490793 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:31.490757    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:41.490428 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:41.490389    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:07:51.491365 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:51.491333    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:07:59.775399 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:59.775363    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:07:59.775804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:59.775647    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" containerID="cri-o://b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee" gracePeriod=30
Jul 28 11:07:59.775804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:07:59.775754    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kube-rbac-proxy" containerID="cri-o://b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee" gracePeriod=30
Jul 28 11:08:00.817948 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:00.817913    2569 generic.go:356] "Generic (PLEG): container finished" podID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerID="b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee" exitCode=2
Jul 28 11:08:00.818342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:00.817988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerDied","Data":"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"}
Jul 28 11:08:01.486154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:01.486112    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 28 11:08:01.490512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:01.490484    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 11:08:04.619392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.619362    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:08:04.730747 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.730709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls\") pod \"074dd06a-204f-48fd-9d11-7f8f7223a226\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") "
Jul 28 11:08:04.730941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.730769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location\") pod \"074dd06a-204f-48fd-9d11-7f8f7223a226\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") "
Jul 28 11:08:04.730941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.730792    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89shv\" (UniqueName: \"kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv\") pod \"074dd06a-204f-48fd-9d11-7f8f7223a226\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") "
Jul 28 11:08:04.730941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.730820    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert\") pod \"074dd06a-204f-48fd-9d11-7f8f7223a226\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") "
Jul 28 11:08:04.730941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.730837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config\") pod \"074dd06a-204f-48fd-9d11-7f8f7223a226\" (UID: \"074dd06a-204f-48fd-9d11-7f8f7223a226\") "
Jul 28 11:08:04.731231 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.731200    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "074dd06a-204f-48fd-9d11-7f8f7223a226" (UID: "074dd06a-204f-48fd-9d11-7f8f7223a226"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:04.731297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.731232    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "074dd06a-204f-48fd-9d11-7f8f7223a226" (UID: "074dd06a-204f-48fd-9d11-7f8f7223a226"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:04.731297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.731236    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "074dd06a-204f-48fd-9d11-7f8f7223a226" (UID: "074dd06a-204f-48fd-9d11-7f8f7223a226"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:04.732838 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.732818    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "074dd06a-204f-48fd-9d11-7f8f7223a226" (UID: "074dd06a-204f-48fd-9d11-7f8f7223a226"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:08:04.732905 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.732867    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv" (OuterVolumeSpecName: "kube-api-access-89shv") pod "074dd06a-204f-48fd-9d11-7f8f7223a226" (UID: "074dd06a-204f-48fd-9d11-7f8f7223a226"). InnerVolumeSpecName "kube-api-access-89shv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:04.831708 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.831667    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/074dd06a-204f-48fd-9d11-7f8f7223a226-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:08:04.831708 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.831711    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89shv\" (UniqueName: \"kubernetes.io/projected/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-api-access-89shv\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:08:04.831999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.831726    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:08:04.831999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.831739    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/074dd06a-204f-48fd-9d11-7f8f7223a226-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:08:04.831999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.831754    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/074dd06a-204f-48fd-9d11-7f8f7223a226-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:08:04.835777 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.835746    2569 generic.go:356] "Generic (PLEG): container finished" podID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerID="b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee" exitCode=0
Jul 28 11:08:04.835915 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.835808    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"
Jul 28 11:08:04.835915 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.835827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerDied","Data":"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"}
Jul 28 11:08:04.835915 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.835897    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx" event={"ID":"074dd06a-204f-48fd-9d11-7f8f7223a226","Type":"ContainerDied","Data":"2431e121842d2a5e05e7e54ab7a268f1b1e99d3a6b2b1fda1640b579c1d344c4"}
Jul 28 11:08:04.836076 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.835936    2569 scope.go:117] "RemoveContainer" containerID="b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"
Jul 28 11:08:04.844054 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.844025    2569 scope.go:117] "RemoveContainer" containerID="b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"
Jul 28 11:08:04.852068 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.852053    2569 scope.go:117] "RemoveContainer" containerID="19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"
Jul 28 11:08:04.859739 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.859716    2569 scope.go:117] "RemoveContainer" containerID="b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"
Jul 28 11:08:04.860010 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:08:04.859994    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee\": container with ID starting with b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee not found: ID does not exist" containerID="b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"
Jul 28 11:08:04.860078 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.860019    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee"} err="failed to get container status \"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee\": rpc error: code = NotFound desc = could not find container \"b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee\": container with ID starting with b328da906992f6e7230ab248a3ea51feeb685dd713d7e3ce1b845ddf36a509ee not found: ID does not exist"
Jul 28 11:08:04.860078 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.860035    2569 scope.go:117] "RemoveContainer" containerID="b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"
Jul 28 11:08:04.860309 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:08:04.860282    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee\": container with ID starting with b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee not found: ID does not exist" containerID="b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"
Jul 28 11:08:04.860425 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.860318    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee"} err="failed to get container status \"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee\": rpc error: code = NotFound desc = could not find container \"b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee\": container with ID starting with b4f70f71da2b26c46c50a2e51566f40e8f1d8f10e38805998359382f6e2bd2ee not found: ID does not exist"
Jul 28 11:08:04.860425 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.860340    2569 scope.go:117] "RemoveContainer" containerID="19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"
Jul 28 11:08:04.860614 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:08:04.860591    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a\": container with ID starting with 19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a not found: ID does not exist" containerID="19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"
Jul 28 11:08:04.860665 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.860624    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a"} err="failed to get container status \"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a\": rpc error: code = NotFound desc = could not find container \"19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a\": container with ID starting with 19a5314a3584b9f567d79dbbf9596f8aab04d1752cc8749d65dc7beb33b4328a not found: ID does not exist"
Jul 28 11:08:04.861247 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.861229    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:08:04.870398 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.870375    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7b2nx"]
Jul 28 11:08:04.956362 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956327    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:08:04.956616 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956604    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kube-rbac-proxy"
Jul 28 11:08:04.956662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956618    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kube-rbac-proxy"
Jul 28 11:08:04.956662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956640    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="storage-initializer"
Jul 28 11:08:04.956662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956647    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="storage-initializer"
Jul 28 11:08:04.956662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956653    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container"
Jul 28 11:08:04.956662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956659    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container"
Jul 28 11:08:04.956846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956716    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kserve-container"
Jul 28 11:08:04.956846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.956724    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" containerName="kube-rbac-proxy"
Jul 28 11:08:04.959613 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.959598    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:04.961994 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.961965    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:08:04.962156 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962144    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 11:08:04.962405 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962384    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:08:04.962514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962416    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:08:04.962514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962474    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 11:08:04.962514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962506    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:08:04.962753 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.962737    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:08:04.969708 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:04.969666    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:08:05.033577 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.033486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.033577 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.033530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.033577 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.033549    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.033833 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.033666    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pnk\" (UniqueName: \"kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.033833 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.033728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.134874 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.134839    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pnk\" (UniqueName: \"kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.134885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.134932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.134959    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.134976    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135272 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:08:05.135088    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 28 11:08:05.135272 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:08:05.135145    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls podName:51568b0e-a4f2-4b67-99a1-3508a4e510e0 nodeName:}" failed. No retries permitted until 2026-07-28 11:08:05.635127404 +0000 UTC m=+3643.555831434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 28 11:08:05.135408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.135385    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135581 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.135562    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.135652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.135636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.144014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.143986    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pnk\" (UniqueName: \"kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.639399 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.639363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.641750 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.641723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.869965 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.869932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:05.999595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:05.999571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:08:06.001918 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:08:06.001895    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51568b0e_a4f2_4b67_99a1_3508a4e510e0.slice/crio-e2d472cefe2dacc67e71555f6978c397e44b3e1c9158959ebed92f7376fdb3b7 WatchSource:0}: Error finding container e2d472cefe2dacc67e71555f6978c397e44b3e1c9158959ebed92f7376fdb3b7: Status 404 returned error can't find the container with id e2d472cefe2dacc67e71555f6978c397e44b3e1c9158959ebed92f7376fdb3b7
Jul 28 11:08:06.566396 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:06.566355    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="074dd06a-204f-48fd-9d11-7f8f7223a226" path="/var/lib/kubelet/pods/074dd06a-204f-48fd-9d11-7f8f7223a226/volumes"
Jul 28 11:08:06.845541 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:06.845455    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerStarted","Data":"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"}
Jul 28 11:08:06.845541 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:06.845511    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerStarted","Data":"e2d472cefe2dacc67e71555f6978c397e44b3e1c9158959ebed92f7376fdb3b7"}
Jul 28 11:08:09.859173 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:09.859138    2569 generic.go:356] "Generic (PLEG): container finished" podID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerID="53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc" exitCode=0
Jul 28 11:08:09.859534 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:09.859213    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerDied","Data":"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"}
Jul 28 11:08:10.866485 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:10.866451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerStarted","Data":"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"}
Jul 28 11:08:10.866906 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:10.866517    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerStarted","Data":"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"}
Jul 28 11:08:10.866906 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:10.866726    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:10.886318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:10.886241    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podStartSLOduration=3.160767025 podStartE2EDuration="6.886224558s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725457533s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:08:10.885994217 +0000 UTC m=+3648.806698271" watchObservedRunningTime="2026-07-28 11:08:10.886224558 +0000 UTC m=+3648.806928610"
Jul 28 11:08:11.870735 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:11.870699    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:11.871952 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:11.871922    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:12.874862 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:12.874824    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:17.878748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:17.878716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:08:17.879377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:17.879348    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:27.879938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:27.879900    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:37.880095 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:37.880051    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:47.879654 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:47.879573    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:08:57.879576 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:08:57.879539    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:09:07.879259 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:07.879222    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:09:17.879596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:17.879553    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:09:18.561813 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:18.561777    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:09:28.565515 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:28.565484    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:09:35.082527 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:35.082496    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:09:35.082928 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:35.082758    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kube-rbac-proxy" containerID="cri-o://e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2" gracePeriod=30
Jul 28 11:09:35.082928 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:35.082746    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" containerID="cri-o://68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c" gracePeriod=30
Jul 28 11:09:35.199451 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:35.199420    2569 generic.go:356] "Generic (PLEG): container finished" podID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerID="e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2" exitCode=2
Jul 28 11:09:35.199611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:35.199490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerDied","Data":"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"}
Jul 28 11:09:37.875512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:37.875466    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jul 28 11:09:38.562190 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:38.562146    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 11:09:39.924863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.924841    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:09:39.930742 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.930722    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location\") pod \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") "
Jul 28 11:09:39.930824 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.930773    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82pnk\" (UniqueName: \"kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk\") pod \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") "
Jul 28 11:09:39.930824 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.930807    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert\") pod \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") "
Jul 28 11:09:39.930927 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.930848    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config\") pod \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") "
Jul 28 11:09:39.930927 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.930875    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") pod \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\" (UID: \"51568b0e-a4f2-4b67-99a1-3508a4e510e0\") "
Jul 28 11:09:39.931107 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.931061    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51568b0e-a4f2-4b67-99a1-3508a4e510e0" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:39.931178 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.931152    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51568b0e-a4f2-4b67-99a1-3508a4e510e0" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:09:39.931241 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.931216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51568b0e-a4f2-4b67-99a1-3508a4e510e0" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:09:39.932794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.932770    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51568b0e-a4f2-4b67-99a1-3508a4e510e0" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:09:39.932880 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:39.932846    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk" (OuterVolumeSpecName: "kube-api-access-82pnk") pod "51568b0e-a4f2-4b67-99a1-3508a4e510e0" (UID: "51568b0e-a4f2-4b67-99a1-3508a4e510e0"). InnerVolumeSpecName "kube-api-access-82pnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:09:40.032228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.032135    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:09:40.032228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.032168    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pnk\" (UniqueName: \"kubernetes.io/projected/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-api-access-82pnk\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:09:40.032228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.032178    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:09:40.032228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.032187    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51568b0e-a4f2-4b67-99a1-3508a4e510e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:09:40.032228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.032197    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51568b0e-a4f2-4b67-99a1-3508a4e510e0-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:09:40.221330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.221294    2569 generic.go:356] "Generic (PLEG): container finished" podID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerID="68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c" exitCode=0
Jul 28 11:09:40.221506 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.221348    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"
Jul 28 11:09:40.221506 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.221378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerDied","Data":"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"}
Jul 28 11:09:40.221506 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.221442    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl" event={"ID":"51568b0e-a4f2-4b67-99a1-3508a4e510e0","Type":"ContainerDied","Data":"e2d472cefe2dacc67e71555f6978c397e44b3e1c9158959ebed92f7376fdb3b7"}
Jul 28 11:09:40.221506 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.221475    2569 scope.go:117] "RemoveContainer" containerID="e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"
Jul 28 11:09:40.229767 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.229746    2569 scope.go:117] "RemoveContainer" containerID="68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"
Jul 28 11:09:40.237605 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.237580    2569 scope.go:117] "RemoveContainer" containerID="53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"
Jul 28 11:09:40.238960 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.238941    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:09:40.239227 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239216    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container"
Jul 28 11:09:40.239272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239229    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container"
Jul 28 11:09:40.239272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239247    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="storage-initializer"
Jul 28 11:09:40.239272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239253    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="storage-initializer"
Jul 28 11:09:40.239272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239261    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kube-rbac-proxy"
Jul 28 11:09:40.239272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239266    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kube-rbac-proxy"
Jul 28 11:09:40.239422 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239315    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kube-rbac-proxy"
Jul 28 11:09:40.239422 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.239324    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" containerName="kserve-container"
Jul 28 11:09:40.243781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.243762    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.245038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245018    2569 scope.go:117] "RemoveContainer" containerID="e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"
Jul 28 11:09:40.245349 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:09:40.245325    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2\": container with ID starting with e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2 not found: ID does not exist" containerID="e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"
Jul 28 11:09:40.245447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245353    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2"} err="failed to get container status \"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2\": rpc error: code = NotFound desc = could not find container \"e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2\": container with ID starting with e649a9e63836c6ca99e6cd1be57e572e7d14eb3e66f5f2681ad60f6fb4f6a0f2 not found: ID does not exist"
Jul 28 11:09:40.245447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245371    2569 scope.go:117] "RemoveContainer" containerID="68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"
Jul 28 11:09:40.245616 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:09:40.245599    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c\": container with ID starting with 68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c not found: ID does not exist" containerID="68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"
Jul 28 11:09:40.245701 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245617    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c"} err="failed to get container status \"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c\": rpc error: code = NotFound desc = could not find container \"68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c\": container with ID starting with 68857aa3124f108dcb93af8b4327243121be3d2529de305a193456ea4fe1935c not found: ID does not exist"
Jul 28 11:09:40.245701 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245630    2569 scope.go:117] "RemoveContainer" containerID="53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"
Jul 28 11:09:40.245861 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:09:40.245845    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc\": container with ID starting with 53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc not found: ID does not exist" containerID="53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"
Jul 28 11:09:40.245918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.245873    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc"} err="failed to get container status \"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc\": rpc error: code = NotFound desc = could not find container \"53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc\": container with ID starting with 53921873e192912c244d4f2b27dbd8b0f81effeb5aa31f2de20baf6f90a770bc not found: ID does not exist"
Jul 28 11:09:40.246496 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246479    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:09:40.246567 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246479    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:09:40.246788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246758    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:09:40.246893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246765    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:09:40.246893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246768    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:09:40.246893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.246827    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 11:09:40.247227 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.247209    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 11:09:40.249636 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.249618    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:09:40.252924 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.252905    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:09:40.256407 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.256387    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-4gwxl"]
Jul 28 11:09:40.335317 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.335227    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.335317 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.335264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.335317 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.335286    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.335553 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.335339    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xzf\" (UniqueName: \"kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.335553 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.335372    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.435860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.435829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436052 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.435885    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436052 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.435906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436052 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.435927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436052 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.435952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xzf\" (UniqueName: \"kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436384 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.436364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436514 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.436494    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.436626 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.436607    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.438377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.438360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.444162 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.444146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xzf\" (UniqueName: \"kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.556272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.556237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:40.569876 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.569842    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51568b0e-a4f2-4b67-99a1-3508a4e510e0" path="/var/lib/kubelet/pods/51568b0e-a4f2-4b67-99a1-3508a4e510e0/volumes"
Jul 28 11:09:40.686706 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:40.686659    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:09:40.688442 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:09:40.688413    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6f6f4e_a2cf_4abc_bbd9_f91a03cd2934.slice/crio-9d89cc36c0593283ed6958bd6493e5dd0dd573d423028c0ab1f625f08346d2dc WatchSource:0}: Error finding container 9d89cc36c0593283ed6958bd6493e5dd0dd573d423028c0ab1f625f08346d2dc: Status 404 returned error can't find the container with id 9d89cc36c0593283ed6958bd6493e5dd0dd573d423028c0ab1f625f08346d2dc
Jul 28 11:09:41.228541 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:41.228504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerStarted","Data":"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"}
Jul 28 11:09:41.228925 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:41.228562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerStarted","Data":"9d89cc36c0593283ed6958bd6493e5dd0dd573d423028c0ab1f625f08346d2dc"}
Jul 28 11:09:45.245194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:45.245163    2569 generic.go:356] "Generic (PLEG): container finished" podID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerID="8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b" exitCode=0
Jul 28 11:09:45.245591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:45.245244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerDied","Data":"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"}
Jul 28 11:09:46.256874 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:46.256832    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerStarted","Data":"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"}
Jul 28 11:09:46.257252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:46.256925    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerStarted","Data":"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"}
Jul 28 11:09:46.257252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:46.257226    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:46.278227 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:46.278157    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podStartSLOduration=2.690257639 podStartE2EDuration="6.278141528s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.587883889s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:46.275478374 +0000 UTC m=+3744.196182425" watchObservedRunningTime="2026-07-28 11:09:46.278141528 +0000 UTC m=+3744.198845581"
Jul 28 11:09:47.260875 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:47.260842    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:09:53.271044 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:09:53.271015    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:10:23.272254 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:10:23.272168    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 11:10:33.272297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:10:33.272254    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 11:10:43.272104 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:10:43.272066    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 11:10:53.272491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:10:53.272450    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 11:11:03.297652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:03.297614    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:11:11.174168 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:11.174131    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:11:11.174585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:11.174383    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" containerID="cri-o://85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef" gracePeriod=30
Jul 28 11:11:11.174585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:11.174427    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kube-rbac-proxy" containerID="cri-o://2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90" gracePeriod=30
Jul 28 11:11:11.588412 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:11.588330    2569 generic.go:356] "Generic (PLEG): container finished" podID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerID="2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90" exitCode=2
Jul 28 11:11:11.588412 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:11.588403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerDied","Data":"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"}
Jul 28 11:11:13.266119 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:13.266079    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Jul 28 11:11:13.271729 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:13.271678    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 11:11:16.026226 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.026199    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:11:16.067387 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067300    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls\") pod \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") "
Jul 28 11:11:16.067387 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067376    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config\") pod \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") "
Jul 28 11:11:16.067593 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067415    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert\") pod \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") "
Jul 28 11:11:16.067593 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location\") pod \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") "
Jul 28 11:11:16.067593 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067454    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5xzf\" (UniqueName: \"kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf\") pod \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\" (UID: \"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934\") "
Jul 28 11:11:16.067822 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067782    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" (UID: "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:11:16.067822 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067817    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" (UID: "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:11:16.067983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.067826    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" (UID: "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:11:16.069491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.069469    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf" (OuterVolumeSpecName: "kube-api-access-l5xzf") pod "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" (UID: "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934"). InnerVolumeSpecName "kube-api-access-l5xzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:11:16.069587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.069518    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" (UID: "1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:11:16.168912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.168878    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:11:16.168912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.168904    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:11:16.168912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.168916    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:11:16.169166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.168926    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xzf\" (UniqueName: \"kubernetes.io/projected/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-kube-api-access-l5xzf\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:11:16.169166 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.168935    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:11:16.390964 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.390855    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:11:16.391187 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kube-rbac-proxy"
Jul 28 11:11:16.391229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391189    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kube-rbac-proxy"
Jul 28 11:11:16.391229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391198    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="storage-initializer"
Jul 28 11:11:16.391229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391204    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="storage-initializer"
Jul 28 11:11:16.391229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391214    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container"
Jul 28 11:11:16.391229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391219    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container"
Jul 28 11:11:16.391385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391287    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kube-rbac-proxy"
Jul 28 11:11:16.391385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.391299    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerName="kserve-container"
Jul 28 11:11:16.394530 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.394513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.397136 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.397111    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 11:11:16.397305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.397162    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 11:11:16.406795 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.406764    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:11:16.471410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.471369    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.471410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.471414    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.471628 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.471474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.471628 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.471507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.471628 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.471554    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxbp\" (UniqueName: \"kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572109    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxbp\" (UniqueName: \"kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572155    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572177    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.572953 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.572928    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.573076 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.573022    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.573076 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.573019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.575066 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.575046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.580876 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.580855    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxbp\" (UniqueName: \"kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.609644 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.609604    2569 generic.go:356] "Generic (PLEG): container finished" podID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" containerID="85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef" exitCode=0
Jul 28 11:11:16.609644 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.609640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerDied","Data":"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"}
Jul 28 11:11:16.609864 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.609659    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"
Jul 28 11:11:16.609864 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.609700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5" event={"ID":"1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934","Type":"ContainerDied","Data":"9d89cc36c0593283ed6958bd6493e5dd0dd573d423028c0ab1f625f08346d2dc"}
Jul 28 11:11:16.609864 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.609727    2569 scope.go:117] "RemoveContainer" containerID="2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"
Jul 28 11:11:16.617801 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.617780    2569 scope.go:117] "RemoveContainer" containerID="85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"
Jul 28 11:11:16.625232 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.625214    2569 scope.go:117] "RemoveContainer" containerID="8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"
Jul 28 11:11:16.626557 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.626535    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:11:16.630885 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.630860    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-xj5v5"]
Jul 28 11:11:16.633250 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.633235    2569 scope.go:117] "RemoveContainer" containerID="2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"
Jul 28 11:11:16.633514 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:11:16.633498    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90\": container with ID starting with 2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90 not found: ID does not exist" containerID="2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"
Jul 28 11:11:16.633562 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.633523    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90"} err="failed to get container status \"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90\": rpc error: code = NotFound desc = could not find container \"2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90\": container with ID starting with 2d80ba6618832a4d4bb5ed1d10963b85453c8f1acacf25b9b58e777519970c90 not found: ID does not exist"
Jul 28 11:11:16.633562 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.633540    2569 scope.go:117] "RemoveContainer" containerID="85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"
Jul 28 11:11:16.633830 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:11:16.633811    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef\": container with ID starting with 85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef not found: ID does not exist" containerID="85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"
Jul 28 11:11:16.633894 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.633838    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef"} err="failed to get container status \"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef\": rpc error: code = NotFound desc = could not find container \"85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef\": container with ID starting with 85bc9bf3600efad2b9bdce64be491a73f50e1e185e28fb87049811c0947455ef not found: ID does not exist"
Jul 28 11:11:16.633894 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.633855    2569 scope.go:117] "RemoveContainer" containerID="8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"
Jul 28 11:11:16.634074 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:11:16.634058    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b\": container with ID starting with 8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b not found: ID does not exist" containerID="8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"
Jul 28 11:11:16.634117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.634078    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b"} err="failed to get container status \"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b\": rpc error: code = NotFound desc = could not find container \"8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b\": container with ID starting with 8ae1a4e228bdc0e70a1a3697c733fc7617c3fbdfc81096d34002bf7c29a2952b not found: ID does not exist"
Jul 28 11:11:16.704595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.704558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:16.837923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:16.837888    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:11:16.840873 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:11:16.840850    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25cbce5_2a17_4ead_8dbe_f56a142d2f20.slice/crio-9ec3192ea3fe4e815aeeb2039b41d20c23017dc06726f4b1974e9abe0469dc38 WatchSource:0}: Error finding container 9ec3192ea3fe4e815aeeb2039b41d20c23017dc06726f4b1974e9abe0469dc38: Status 404 returned error can't find the container with id 9ec3192ea3fe4e815aeeb2039b41d20c23017dc06726f4b1974e9abe0469dc38
Jul 28 11:11:17.617939 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:17.617900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerStarted","Data":"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"}
Jul 28 11:11:17.618357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:17.617959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerStarted","Data":"9ec3192ea3fe4e815aeeb2039b41d20c23017dc06726f4b1974e9abe0469dc38"}
Jul 28 11:11:18.565931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:18.565896    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934" path="/var/lib/kubelet/pods/1c6f6f4e-a2cf-4abc-bbd9-f91a03cd2934/volumes"
Jul 28 11:11:21.634754 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:21.634714    2569 generic.go:356] "Generic (PLEG): container finished" podID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerID="f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959" exitCode=0
Jul 28 11:11:21.635256 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:21.634775    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerDied","Data":"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"}
Jul 28 11:11:22.641429 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:22.641395    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerStarted","Data":"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"}
Jul 28 11:11:22.641856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:22.641457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerStarted","Data":"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"}
Jul 28 11:11:22.641856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:22.641772    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:22.641856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:22.641821    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:22.662126 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:22.662078    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podStartSLOduration=2.674549802 podStartE2EDuration="6.662061819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.987512017s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:11:22.661537089 +0000 UTC m=+3840.582241142" watchObservedRunningTime="2026-07-28 11:11:22.662061819 +0000 UTC m=+3840.582765870"
Jul 28 11:11:28.651102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:28.651070    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:11:58.652612 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:11:58.652572    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 11:12:08.652459 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:08.652420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 11:12:18.652524 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:18.652485    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 11:12:22.720525 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:22.720490    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:12:22.732841 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:22.732819    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:12:28.652058 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:28.652016    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 11:12:31.561963 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:31.561921    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.61:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 11:12:41.597382 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:41.597346    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:12:46.448414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:46.448376    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:12:46.448865 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:46.448663    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container" containerID="cri-o://8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7" gracePeriod=30
Jul 28 11:12:46.448865 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:46.448711    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kube-rbac-proxy" containerID="cri-o://fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8" gracePeriod=30
Jul 28 11:12:46.968140 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:46.968107    2569 generic.go:356] "Generic (PLEG): container finished" podID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerID="fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8" exitCode=2
Jul 28 11:12:46.968312 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:46.968163    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerDied","Data":"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"}
Jul 28 11:12:48.646643 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:48.646599    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 11:12:51.195338 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.195311    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:12:51.267024 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.266940    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rxbp\" (UniqueName: \"kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp\") pod \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") "
Jul 28 11:12:51.267024 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267009    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert\") pod \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") "
Jul 28 11:12:51.267253 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267029    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config\") pod \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") "
Jul 28 11:12:51.267253 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267060    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls\") pod \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") "
Jul 28 11:12:51.267253 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267078    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location\") pod \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\" (UID: \"d25cbce5-2a17-4ead-8dbe-f56a142d2f20\") "
Jul 28 11:12:51.267430 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267407    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d25cbce5-2a17-4ead-8dbe-f56a142d2f20" (UID: "d25cbce5-2a17-4ead-8dbe-f56a142d2f20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:12:51.267498 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267407    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d25cbce5-2a17-4ead-8dbe-f56a142d2f20" (UID: "d25cbce5-2a17-4ead-8dbe-f56a142d2f20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:12:51.267498 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.267453    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d25cbce5-2a17-4ead-8dbe-f56a142d2f20" (UID: "d25cbce5-2a17-4ead-8dbe-f56a142d2f20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:12:51.269077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.269058    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d25cbce5-2a17-4ead-8dbe-f56a142d2f20" (UID: "d25cbce5-2a17-4ead-8dbe-f56a142d2f20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:12:51.269167 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.269149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp" (OuterVolumeSpecName: "kube-api-access-9rxbp") pod "d25cbce5-2a17-4ead-8dbe-f56a142d2f20" (UID: "d25cbce5-2a17-4ead-8dbe-f56a142d2f20"). InnerVolumeSpecName "kube-api-access-9rxbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:12:51.368469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.368418    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:12:51.368469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.368462    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:12:51.368469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.368473    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:12:51.368469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.368481    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:12:51.368469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.368490    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rxbp\" (UniqueName: \"kubernetes.io/projected/d25cbce5-2a17-4ead-8dbe-f56a142d2f20-kube-api-access-9rxbp\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:12:51.608370 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608293    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:12:51.608615 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608601    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container"
Jul 28 11:12:51.608659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608617    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container"
Jul 28 11:12:51.608659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608627    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kube-rbac-proxy"
Jul 28 11:12:51.608659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608634    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kube-rbac-proxy"
Jul 28 11:12:51.608659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608644    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="storage-initializer"
Jul 28 11:12:51.608659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608649    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="storage-initializer"
Jul 28 11:12:51.608841 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608726    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kserve-container"
Jul 28 11:12:51.608841 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.608734    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerName="kube-rbac-proxy"
Jul 28 11:12:51.611955 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.611939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.614120 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.614094    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 11:12:51.614252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.614124    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 11:12:51.622382 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.622353    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:12:51.669924 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.669885    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8j7m\" (UniqueName: \"kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.670108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.669933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.670108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.669965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.670108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.669988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.670108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.670003    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.770945 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.770913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8j7m\" (UniqueName: \"kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771148 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.770960    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771148 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.770985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771148 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.771007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771148 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.771024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771397 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.771378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771656 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.771632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.771793 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.771723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.773432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.773416    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.778901 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.778879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8j7m\" (UniqueName: \"kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.923335 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.923291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:51.990329 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.990295    2569 generic.go:356] "Generic (PLEG): container finished" podID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" containerID="8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7" exitCode=0
Jul 28 11:12:51.990512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.990346    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"
Jul 28 11:12:51.990512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.990378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerDied","Data":"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"}
Jul 28 11:12:51.990512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.990447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n" event={"ID":"d25cbce5-2a17-4ead-8dbe-f56a142d2f20","Type":"ContainerDied","Data":"9ec3192ea3fe4e815aeeb2039b41d20c23017dc06726f4b1974e9abe0469dc38"}
Jul 28 11:12:51.990512 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:51.990484    2569 scope.go:117] "RemoveContainer" containerID="fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"
Jul 28 11:12:52.012954 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.012935    2569 scope.go:117] "RemoveContainer" containerID="8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"
Jul 28 11:12:52.020719 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.020695    2569 scope.go:117] "RemoveContainer" containerID="f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"
Jul 28 11:12:52.024928 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.024883    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:12:52.029837 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.029744    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-jhx7n"]
Jul 28 11:12:52.029910 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.029877    2569 scope.go:117] "RemoveContainer" containerID="fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"
Jul 28 11:12:52.030193 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:12:52.030164    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8\": container with ID starting with fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8 not found: ID does not exist" containerID="fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"
Jul 28 11:12:52.030252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.030200    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8"} err="failed to get container status \"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8\": rpc error: code = NotFound desc = could not find container \"fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8\": container with ID starting with fc95fc9570b94d39b520812b75c60ddedcd3ba11e78d8eafefcc7969443cd1e8 not found: ID does not exist"
Jul 28 11:12:52.030252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.030226    2569 scope.go:117] "RemoveContainer" containerID="8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"
Jul 28 11:12:52.030520 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:12:52.030496    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7\": container with ID starting with 8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7 not found: ID does not exist" containerID="8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"
Jul 28 11:12:52.030572 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.030529    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7"} err="failed to get container status \"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7\": rpc error: code = NotFound desc = could not find container \"8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7\": container with ID starting with 8bdfa3ce65cc0bbe5dfda6fb36434be5454dfb25116868b4bd21914de4d555d7 not found: ID does not exist"
Jul 28 11:12:52.030572 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.030552    2569 scope.go:117] "RemoveContainer" containerID="f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"
Jul 28 11:12:52.030875 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:12:52.030822    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959\": container with ID starting with f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959 not found: ID does not exist" containerID="f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"
Jul 28 11:12:52.030875 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.030852    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959"} err="failed to get container status \"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959\": rpc error: code = NotFound desc = could not find container \"f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959\": container with ID starting with f4f136cf19c37011d04ee2c8980a634d3305248e6deb557c93588b51fd852959 not found: ID does not exist"
Jul 28 11:12:52.061645 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.061623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:12:52.064059 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:12:52.064034    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb30123_ce84_40ad_b9b6_fd93410dd0c2.slice/crio-0708d215ca85633da7c43639ea113627c2db09dd7b0d3a86792b5a480005c06a WatchSource:0}: Error finding container 0708d215ca85633da7c43639ea113627c2db09dd7b0d3a86792b5a480005c06a: Status 404 returned error can't find the container with id 0708d215ca85633da7c43639ea113627c2db09dd7b0d3a86792b5a480005c06a
Jul 28 11:12:52.565877 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.565844    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25cbce5-2a17-4ead-8dbe-f56a142d2f20" path="/var/lib/kubelet/pods/d25cbce5-2a17-4ead-8dbe-f56a142d2f20/volumes"
Jul 28 11:12:52.997025 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.996991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerStarted","Data":"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"}
Jul 28 11:12:52.997254 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:52.997056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerStarted","Data":"0708d215ca85633da7c43639ea113627c2db09dd7b0d3a86792b5a480005c06a"}
Jul 28 11:12:56.010820 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:56.010786    2569 generic.go:356] "Generic (PLEG): container finished" podID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerID="400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6" exitCode=0
Jul 28 11:12:56.011201 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:56.010848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerDied","Data":"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"}
Jul 28 11:12:57.016956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:57.016917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerStarted","Data":"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"}
Jul 28 11:12:57.017383 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:57.017012    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerStarted","Data":"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"}
Jul 28 11:12:57.017383 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:57.017225    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:57.017383 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:57.017283    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:12:57.038550 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:12:57.038499    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podStartSLOduration=2.245716458 podStartE2EDuration="6.038483107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792766649s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:12:57.037977171 +0000 UTC m=+3934.958681225" watchObservedRunningTime="2026-07-28 11:12:57.038483107 +0000 UTC m=+3934.959187159"
Jul 28 11:13:03.027224 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:13:03.027194    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:13:33.028609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:13:33.028567    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 11:13:43.028675 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:13:43.028636    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 11:13:53.027762 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:13:53.027717    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 11:14:03.028390 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:03.028345    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 11:14:13.096846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:13.096814    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:14:21.724128 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:21.724095    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:14:21.724600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:21.724419    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kube-rbac-proxy" containerID="cri-o://249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81" gracePeriod=30
Jul 28 11:14:21.724600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:21.724525    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" containerID="cri-o://378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25" gracePeriod=30
Jul 28 11:14:22.355297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:22.355265    2569 generic.go:356] "Generic (PLEG): container finished" podID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerID="249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81" exitCode=2
Jul 28 11:14:22.355491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:22.355338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerDied","Data":"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"}
Jul 28 11:14:23.021998 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:23.021963    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Jul 28 11:14:23.028552 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:23.028513    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 11:14:26.867796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.867770    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:14:26.981019 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.980983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8j7m\" (UniqueName: \"kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m\") pod \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") "
Jul 28 11:14:26.981207 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981033    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert\") pod \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") "
Jul 28 11:14:26.981207 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981071    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location\") pod \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") "
Jul 28 11:14:26.981207 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls\") pod \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") "
Jul 28 11:14:26.981207 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981144    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config\") pod \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\" (UID: \"afb30123-ce84-40ad-b9b6-fd93410dd0c2\") "
Jul 28 11:14:26.981493 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981467    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afb30123-ce84-40ad-b9b6-fd93410dd0c2" (UID: "afb30123-ce84-40ad-b9b6-fd93410dd0c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:14:26.981554 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981471    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afb30123-ce84-40ad-b9b6-fd93410dd0c2" (UID: "afb30123-ce84-40ad-b9b6-fd93410dd0c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:14:26.981554 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.981524    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afb30123-ce84-40ad-b9b6-fd93410dd0c2" (UID: "afb30123-ce84-40ad-b9b6-fd93410dd0c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:14:26.983201 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.983171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afb30123-ce84-40ad-b9b6-fd93410dd0c2" (UID: "afb30123-ce84-40ad-b9b6-fd93410dd0c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:14:26.983201 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:26.983194    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m" (OuterVolumeSpecName: "kube-api-access-s8j7m") pod "afb30123-ce84-40ad-b9b6-fd93410dd0c2" (UID: "afb30123-ce84-40ad-b9b6-fd93410dd0c2"). InnerVolumeSpecName "kube-api-access-s8j7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:14:27.082510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.082327    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afb30123-ce84-40ad-b9b6-fd93410dd0c2-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:14:27.082510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.082365    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:14:27.082510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.082380    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8j7m\" (UniqueName: \"kubernetes.io/projected/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kube-api-access-s8j7m\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:14:27.082510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.082394    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afb30123-ce84-40ad-b9b6-fd93410dd0c2-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:14:27.082510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.082408    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afb30123-ce84-40ad-b9b6-fd93410dd0c2-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:14:27.376473 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.376384    2569 generic.go:356] "Generic (PLEG): container finished" podID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerID="378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25" exitCode=0
Jul 28 11:14:27.376473 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.376444    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"
Jul 28 11:14:27.376739 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.376468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerDied","Data":"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"}
Jul 28 11:14:27.376739 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.376540    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm" event={"ID":"afb30123-ce84-40ad-b9b6-fd93410dd0c2","Type":"ContainerDied","Data":"0708d215ca85633da7c43639ea113627c2db09dd7b0d3a86792b5a480005c06a"}
Jul 28 11:14:27.376739 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.376576    2569 scope.go:117] "RemoveContainer" containerID="249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"
Jul 28 11:14:27.384803 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.384782    2569 scope.go:117] "RemoveContainer" containerID="378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"
Jul 28 11:14:27.391843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.391824    2569 scope.go:117] "RemoveContainer" containerID="400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"
Jul 28 11:14:27.397918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.397896    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:14:27.399028 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399015    2569 scope.go:117] "RemoveContainer" containerID="249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"
Jul 28 11:14:27.399284 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:14:27.399263    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81\": container with ID starting with 249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81 not found: ID does not exist" containerID="249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"
Jul 28 11:14:27.399339 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399293    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81"} err="failed to get container status \"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81\": rpc error: code = NotFound desc = could not find container \"249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81\": container with ID starting with 249dd81fe834c7836b72911a8220169d236c79a62f9387d1d1a51a6f848bdc81 not found: ID does not exist"
Jul 28 11:14:27.399339 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399311    2569 scope.go:117] "RemoveContainer" containerID="378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"
Jul 28 11:14:27.399550 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:14:27.399534    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25\": container with ID starting with 378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25 not found: ID does not exist" containerID="378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"
Jul 28 11:14:27.399604 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399556    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25"} err="failed to get container status \"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25\": rpc error: code = NotFound desc = could not find container \"378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25\": container with ID starting with 378bd2aa3836e179106959910ae83f57b29b3951d8bc5ba8ffbf736acb679d25 not found: ID does not exist"
Jul 28 11:14:27.399604 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399569    2569 scope.go:117] "RemoveContainer" containerID="400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"
Jul 28 11:14:27.399773 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:14:27.399755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6\": container with ID starting with 400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6 not found: ID does not exist" containerID="400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"
Jul 28 11:14:27.399821 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.399775    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6"} err="failed to get container status \"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6\": rpc error: code = NotFound desc = could not find container \"400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6\": container with ID starting with 400a8135913053efd38800da401788085af8e481a64818ed995cefb6c25c93f6 not found: ID does not exist"
Jul 28 11:14:27.402465 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:27.402445    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-lddpm"]
Jul 28 11:14:28.565415 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:28.565384    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" path="/var/lib/kubelet/pods/afb30123-ce84-40ad-b9b6-fd93410dd0c2/volumes"
Jul 28 11:14:34.019535 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019498    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kube-rbac-proxy"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019835    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kube-rbac-proxy"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019848    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019854    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019874    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="storage-initializer"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019880    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="storage-initializer"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019927    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kserve-container"
Jul 28 11:14:34.020031 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.019937    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb30123-ce84-40ad-b9b6-fd93410dd0c2" containerName="kube-rbac-proxy"
Jul 28 11:14:34.024338 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.024322    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.026983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.026949    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:14:34.026983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.026962    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:14:34.026983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.026953    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 11:14:34.026983 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.026983    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:14:34.027273 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.027003    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:14:34.027976 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.027955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 11:14:34.027976 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.027955    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:14:34.033224 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.033203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:14:34.138647 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.138604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.138847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.138657    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.138847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.138780    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkgs\" (UniqueName: \"kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.138847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.138825    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.138975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.138888    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkgs\" (UniqueName: \"kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239676 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239509    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.239996 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.239974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.240188 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.240170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.240244 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.240228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.241969 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.241951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.247926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.247893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkgs\" (UniqueName: \"kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs\") pod \"isvc-sklearn-predictor-84c77c5fdf-srbxm\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.334934 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.334831    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:34.466322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:34.466239    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:14:34.468960 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:14:34.468929    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b3ee93_c024_424f_8a69_4fab8583e2d6.slice/crio-853720a0c5d170e1ea7381f5e8c988e96e5b56cd12f310297b9863461db42a5d WatchSource:0}: Error finding container 853720a0c5d170e1ea7381f5e8c988e96e5b56cd12f310297b9863461db42a5d: Status 404 returned error can't find the container with id 853720a0c5d170e1ea7381f5e8c988e96e5b56cd12f310297b9863461db42a5d
Jul 28 11:14:35.418238 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:35.418196    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerStarted","Data":"c3dc4b6010a09e37d0b5bc65b2320513076e948b36a16b4076d33c24c4e83afd"}
Jul 28 11:14:35.418651 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:35.418269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerStarted","Data":"853720a0c5d170e1ea7381f5e8c988e96e5b56cd12f310297b9863461db42a5d"}
Jul 28 11:14:38.432254 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:38.432217    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerID="c3dc4b6010a09e37d0b5bc65b2320513076e948b36a16b4076d33c24c4e83afd" exitCode=0
Jul 28 11:14:38.432657 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:38.432291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerDied","Data":"c3dc4b6010a09e37d0b5bc65b2320513076e948b36a16b4076d33c24c4e83afd"}
Jul 28 11:14:39.438354 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.438320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerStarted","Data":"61a192067663ab9c96a1c77a1a768845efbc08efce8f50e9b19de158d21c2cc9"}
Jul 28 11:14:39.438775 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.438382    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerStarted","Data":"d0c536b55a4d146e39cd78c3891c80952e7b1b2073c4b8e2c3108a2e5584c17a"}
Jul 28 11:14:39.438775 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.438648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:39.438898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.438801    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:39.440021 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.439994    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:14:39.460735 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:39.460667    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podStartSLOduration=1.85620772 podStartE2EDuration="5.460651814s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604444094s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:14:39.459167235 +0000 UTC m=+4037.379871287" watchObservedRunningTime="2026-07-28 11:14:39.460651814 +0000 UTC m=+4037.381355866"
Jul 28 11:14:40.442890 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:40.442854    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:14:45.447985 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:45.447904    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:14:45.448415 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:45.448380    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:14:55.448998 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:14:55.448959    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:05.448443 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:05.448398    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:15.448436 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:15.448396    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:25.449371 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:25.449332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:35.449043 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:35.449006    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:45.448884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:45.448839    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:15:55.449304 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:15:55.449266    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:16:04.189167 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:04.189128    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:16:04.189596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:04.189391    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" containerID="cri-o://d0c536b55a4d146e39cd78c3891c80952e7b1b2073c4b8e2c3108a2e5584c17a" gracePeriod=30
Jul 28 11:16:04.189596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:04.189430    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kube-rbac-proxy" containerID="cri-o://61a192067663ab9c96a1c77a1a768845efbc08efce8f50e9b19de158d21c2cc9" gracePeriod=30
Jul 28 11:16:04.787111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:04.787080    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerID="61a192067663ab9c96a1c77a1a768845efbc08efce8f50e9b19de158d21c2cc9" exitCode=2
Jul 28 11:16:04.787292 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:04.787151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerDied","Data":"61a192067663ab9c96a1c77a1a768845efbc08efce8f50e9b19de158d21c2cc9"}
Jul 28 11:16:05.443493 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:05.443444    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Jul 28 11:16:05.448844 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:05.448818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 11:16:08.807367 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:08.807331    2569 generic.go:356] "Generic (PLEG): container finished" podID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerID="d0c536b55a4d146e39cd78c3891c80952e7b1b2073c4b8e2c3108a2e5584c17a" exitCode=0
Jul 28 11:16:08.807756 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:08.807391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerDied","Data":"d0c536b55a4d146e39cd78c3891c80952e7b1b2073c4b8e2c3108a2e5584c17a"}
Jul 28 11:16:08.923803 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:08.923774    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:16:09.013595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013508    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert\") pod \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") "
Jul 28 11:16:09.013595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013546    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config\") pod \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") "
Jul 28 11:16:09.013595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013583    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location\") pod \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") "
Jul 28 11:16:09.013930 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls\") pod \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") "
Jul 28 11:16:09.013930 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013632    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbkgs\" (UniqueName: \"kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs\") pod \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\" (UID: \"f2b3ee93-c024-424f-8a69-4fab8583e2d6\") "
Jul 28 11:16:09.014036 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013955    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2b3ee93-c024-424f-8a69-4fab8583e2d6" (UID: "f2b3ee93-c024-424f-8a69-4fab8583e2d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:16:09.014036 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.013973    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2b3ee93-c024-424f-8a69-4fab8583e2d6" (UID: "f2b3ee93-c024-424f-8a69-4fab8583e2d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:16:09.014110 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.014036    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2b3ee93-c024-424f-8a69-4fab8583e2d6" (UID: "f2b3ee93-c024-424f-8a69-4fab8583e2d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:16:09.015681 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.015654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs" (OuterVolumeSpecName: "kube-api-access-jbkgs") pod "f2b3ee93-c024-424f-8a69-4fab8583e2d6" (UID: "f2b3ee93-c024-424f-8a69-4fab8583e2d6"). InnerVolumeSpecName "kube-api-access-jbkgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:16:09.015681 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.015664    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2b3ee93-c024-424f-8a69-4fab8583e2d6" (UID: "f2b3ee93-c024-424f-8a69-4fab8583e2d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:16:09.115297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.115250    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:16:09.115297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.115293    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:16:09.115297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.115303    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:16:09.115297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.115311    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2b3ee93-c024-424f-8a69-4fab8583e2d6-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:16:09.115583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.115320    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbkgs\" (UniqueName: \"kubernetes.io/projected/f2b3ee93-c024-424f-8a69-4fab8583e2d6-kube-api-access-jbkgs\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:16:09.351039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.350952    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351613    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kube-rbac-proxy"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351638    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kube-rbac-proxy"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351656    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="storage-initializer"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351742    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="storage-initializer"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351791    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.351811    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.352014    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kube-rbac-proxy"
Jul 28 11:16:09.352420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.352034    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" containerName="kserve-container"
Jul 28 11:16:09.356774 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.356748    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.359660 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.359634    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 11:16:09.359782 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.359672    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 11:16:09.365555 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.365527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:16:09.417323 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.417291    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp5j\" (UniqueName: \"kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.417502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.417337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.417502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.417366    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.417502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.417405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.417502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.417437    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.518527 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.518492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp5j\" (UniqueName: \"kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.518741 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.518536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.518741 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.518555    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.518741 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.518591    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.518741 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.518616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.519032 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.519010    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.519411 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.519384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.519521 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.519384    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.521084 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.521067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.526568 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.526549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp5j\" (UniqueName: \"kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.668103 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.668065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:09.797957 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.797928    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:16:09.800615 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:16:09.800588    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c08fbe8_2758_496c_8373_45ecdb78b813.slice/crio-e4707c4836eace6005a6fcef95e3aeb3b60bccb77b06fd8cb181dd8d016de5c7 WatchSource:0}: Error finding container e4707c4836eace6005a6fcef95e3aeb3b60bccb77b06fd8cb181dd8d016de5c7: Status 404 returned error can't find the container with id e4707c4836eace6005a6fcef95e3aeb3b60bccb77b06fd8cb181dd8d016de5c7
Jul 28 11:16:09.812139 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.812109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerStarted","Data":"e4707c4836eace6005a6fcef95e3aeb3b60bccb77b06fd8cb181dd8d016de5c7"}
Jul 28 11:16:09.813938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.813915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm" event={"ID":"f2b3ee93-c024-424f-8a69-4fab8583e2d6","Type":"ContainerDied","Data":"853720a0c5d170e1ea7381f5e8c988e96e5b56cd12f310297b9863461db42a5d"}
Jul 28 11:16:09.814048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.813966    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"
Jul 28 11:16:09.814048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.813984    2569 scope.go:117] "RemoveContainer" containerID="61a192067663ab9c96a1c77a1a768845efbc08efce8f50e9b19de158d21c2cc9"
Jul 28 11:16:09.825702 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.825663    2569 scope.go:117] "RemoveContainer" containerID="d0c536b55a4d146e39cd78c3891c80952e7b1b2073c4b8e2c3108a2e5584c17a"
Jul 28 11:16:09.842474 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.842453    2569 scope.go:117] "RemoveContainer" containerID="c3dc4b6010a09e37d0b5bc65b2320513076e948b36a16b4076d33c24c4e83afd"
Jul 28 11:16:09.852016 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.851994    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:16:09.859580 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:09.859553    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-84c77c5fdf-srbxm"]
Jul 28 11:16:10.565937 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:10.565903    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b3ee93-c024-424f-8a69-4fab8583e2d6" path="/var/lib/kubelet/pods/f2b3ee93-c024-424f-8a69-4fab8583e2d6/volumes"
Jul 28 11:16:10.819652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:10.819557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerStarted","Data":"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"}
Jul 28 11:16:13.834802 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:13.834710    2569 generic.go:356] "Generic (PLEG): container finished" podID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerID="3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d" exitCode=0
Jul 28 11:16:13.835259 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:13.834804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerDied","Data":"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"}
Jul 28 11:16:14.841671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:14.841591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerStarted","Data":"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"}
Jul 28 11:16:14.841671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:14.841657    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerStarted","Data":"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"}
Jul 28 11:16:14.842172 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:14.841879    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:14.864139 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:14.864090    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podStartSLOduration=2.170429276 podStartE2EDuration="5.864072792s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.693643516s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:16:14.8626392 +0000 UTC m=+4132.783343235" watchObservedRunningTime="2026-07-28 11:16:14.864072792 +0000 UTC m=+4132.784776844"
Jul 28 11:16:15.846029 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:15.845999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:21.856109 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:21.856080    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:16:51.893752 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:16:51.893697    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 11:17:01.859039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:01.859004    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:17:09.419499 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:09.419464    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:17:09.419906 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:09.419726    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container" containerID="cri-o://c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec" gracePeriod=30
Jul 28 11:17:09.419906 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:09.419760    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy" containerID="cri-o://34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d" gracePeriod=30
Jul 28 11:17:10.061185 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:10.061153    2569 generic.go:356] "Generic (PLEG): container finished" podID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerID="34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d" exitCode=2
Jul 28 11:17:10.061371 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:10.061222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerDied","Data":"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"}
Jul 28 11:17:11.851507 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:11.851460    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Jul 28 11:17:12.897895 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:12.897850    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 11:17:14.581886 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.581846    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:14.585307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.585291    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.587982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.587957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 11:17:14.587982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.587968    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 11:17:14.596094 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.596068    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:14.680163 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.680119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md95p\" (UniqueName: \"kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.680163 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.680170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.680539 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.680513    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.680646 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.680587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.680757 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.680640    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782107 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782329 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782329 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782329 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782175    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782329 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md95p\" (UniqueName: \"kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782714 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782951 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782918    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.782951 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.782933    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.784742 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.784723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.790432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.790401    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md95p\" (UniqueName: \"kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p\") pod \"isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:14.896640 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:14.896612    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:15.033167 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:15.033141    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:15.035341 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:17:15.035311    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a13896_e1c7_4253_8381_db3fee1846f1.slice/crio-0d8077698f3a60c65dc4b9775b67d308317dc03b3a27462d59f13fca890b12cf WatchSource:0}: Error finding container 0d8077698f3a60c65dc4b9775b67d308317dc03b3a27462d59f13fca890b12cf: Status 404 returned error can't find the container with id 0d8077698f3a60c65dc4b9775b67d308317dc03b3a27462d59f13fca890b12cf
Jul 28 11:17:15.082590 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:15.082555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerStarted","Data":"0d8077698f3a60c65dc4b9775b67d308317dc03b3a27462d59f13fca890b12cf"}
Jul 28 11:17:16.088125 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:16.088087    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerStarted","Data":"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"}
Jul 28 11:17:16.851231 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:16.851185    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Jul 28 11:17:17.059386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.059363    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:17:17.094482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.094442    2569 generic.go:356] "Generic (PLEG): container finished" podID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerID="c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec" exitCode=0
Jul 28 11:17:17.094898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.094498    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"
Jul 28 11:17:17.094898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.094522    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerDied","Data":"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"}
Jul 28 11:17:17.094898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.094599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c" event={"ID":"7c08fbe8-2758-496c-8373-45ecdb78b813","Type":"ContainerDied","Data":"e4707c4836eace6005a6fcef95e3aeb3b60bccb77b06fd8cb181dd8d016de5c7"}
Jul 28 11:17:17.094898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.094659    2569 scope.go:117] "RemoveContainer" containerID="34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"
Jul 28 11:17:17.102797 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.102766    2569 scope.go:117] "RemoveContainer" containerID="c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"
Jul 28 11:17:17.110441 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.110415    2569 scope.go:117] "RemoveContainer" containerID="3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"
Jul 28 11:17:17.117672 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.117656    2569 scope.go:117] "RemoveContainer" containerID="34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"
Jul 28 11:17:17.117983 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:17.117960    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d\": container with ID starting with 34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d not found: ID does not exist" containerID="34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"
Jul 28 11:17:17.118047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.117990    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d"} err="failed to get container status \"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d\": rpc error: code = NotFound desc = could not find container \"34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d\": container with ID starting with 34aa0b8d6226861d10c8e1f783c01c84e9330b2bf8e7334e846ee84316d9c31d not found: ID does not exist"
Jul 28 11:17:17.118047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.118007    2569 scope.go:117] "RemoveContainer" containerID="c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"
Jul 28 11:17:17.118245 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:17.118228    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec\": container with ID starting with c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec not found: ID does not exist" containerID="c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"
Jul 28 11:17:17.118287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.118249    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec"} err="failed to get container status \"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec\": rpc error: code = NotFound desc = could not find container \"c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec\": container with ID starting with c2de7eb721ad7f168afb4e1819b827ca0383e15f495d361448e95531fe564aec not found: ID does not exist"
Jul 28 11:17:17.118287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.118262    2569 scope.go:117] "RemoveContainer" containerID="3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"
Jul 28 11:17:17.118503 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:17.118487    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d\": container with ID starting with 3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d not found: ID does not exist" containerID="3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"
Jul 28 11:17:17.118552 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.118513    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d"} err="failed to get container status \"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d\": rpc error: code = NotFound desc = could not find container \"3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d\": container with ID starting with 3d9486d5146af778384825282500eb7f68c7ed87d155a59068c99235fe84221d not found: ID does not exist"
Jul 28 11:17:17.204909 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.204872    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config\") pod \"7c08fbe8-2758-496c-8373-45ecdb78b813\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") "
Jul 28 11:17:17.205057 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.204935    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls\") pod \"7c08fbe8-2758-496c-8373-45ecdb78b813\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") "
Jul 28 11:17:17.205057 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.204963    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmp5j\" (UniqueName: \"kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j\") pod \"7c08fbe8-2758-496c-8373-45ecdb78b813\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") "
Jul 28 11:17:17.205057 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.204998    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert\") pod \"7c08fbe8-2758-496c-8373-45ecdb78b813\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") "
Jul 28 11:17:17.205057 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.205042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location\") pod \"7c08fbe8-2758-496c-8373-45ecdb78b813\" (UID: \"7c08fbe8-2758-496c-8373-45ecdb78b813\") "
Jul 28 11:17:17.205330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.205299    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c08fbe8-2758-496c-8373-45ecdb78b813" (UID: "7c08fbe8-2758-496c-8373-45ecdb78b813"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:17:17.205482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.205457    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c08fbe8-2758-496c-8373-45ecdb78b813" (UID: "7c08fbe8-2758-496c-8373-45ecdb78b813"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:17:17.205625 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.205481    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c08fbe8-2758-496c-8373-45ecdb78b813" (UID: "7c08fbe8-2758-496c-8373-45ecdb78b813"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:17.207658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.207637    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c08fbe8-2758-496c-8373-45ecdb78b813" (UID: "7c08fbe8-2758-496c-8373-45ecdb78b813"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:17:17.207658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.207650    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j" (OuterVolumeSpecName: "kube-api-access-pmp5j") pod "7c08fbe8-2758-496c-8373-45ecdb78b813" (UID: "7c08fbe8-2758-496c-8373-45ecdb78b813"). InnerVolumeSpecName "kube-api-access-pmp5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:17:17.306111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.306078    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c08fbe8-2758-496c-8373-45ecdb78b813-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:17.306111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.306106    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:17.306111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.306116    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c08fbe8-2758-496c-8373-45ecdb78b813-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:17.306349 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.306127    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp5j\" (UniqueName: \"kubernetes.io/projected/7c08fbe8-2758-496c-8373-45ecdb78b813-kube-api-access-pmp5j\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:17.306349 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.306136    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c08fbe8-2758-496c-8373-45ecdb78b813-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:17.418098 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.418069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:17:17.422096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:17.422069    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2bq2c"]
Jul 28 11:17:18.566126 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:18.566094    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" path="/var/lib/kubelet/pods/7c08fbe8-2758-496c-8373-45ecdb78b813/volumes"
Jul 28 11:17:21.114694 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:21.114648    2569 generic.go:356] "Generic (PLEG): container finished" podID="72a13896-e1c7-4253-8381-db3fee1846f1" containerID="7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441" exitCode=0
Jul 28 11:17:21.115089 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:21.114715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerDied","Data":"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"}
Jul 28 11:17:22.122022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.121986    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerStarted","Data":"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"}
Jul 28 11:17:22.122472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.122047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerStarted","Data":"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"}
Jul 28 11:17:22.122472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.122333    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:22.122603 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.122496    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:22.123621 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.123601    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Jul 28 11:17:22.144661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.144620    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podStartSLOduration=2.163973269 podStartE2EDuration="8.144602954s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.980629685s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:17:22.1426073 +0000 UTC m=+4200.063311353" watchObservedRunningTime="2026-07-28 11:17:22.144602954 +0000 UTC m=+4200.065307005"
Jul 28 11:17:22.753026 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.752987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:17:22.766582 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:22.766557    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:17:23.127157 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:23.127068    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Jul 28 11:17:28.131338 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:28.131308    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:28.131881 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:28.131854    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Jul 28 11:17:38.132860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:38.132828    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:46.415297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:46.415226    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn_72a13896-e1c7-4253-8381-db3fee1846f1/kserve-container/0.log"
Jul 28 11:17:46.616531 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:46.616488    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:46.616804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:46.616775    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container" containerID="cri-o://59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53" gracePeriod=30
Jul 28 11:17:46.616947 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:46.616798    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kube-rbac-proxy" containerID="cri-o://75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb" gracePeriod=30
Jul 28 11:17:47.232622 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.232589    2569 generic.go:356] "Generic (PLEG): container finished" podID="72a13896-e1c7-4253-8381-db3fee1846f1" containerID="75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb" exitCode=2
Jul 28 11:17:47.232826 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.232665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerDied","Data":"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"}
Jul 28 11:17:47.464975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.464949    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:47.561462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location\") pod \"72a13896-e1c7-4253-8381-db3fee1846f1\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") "
Jul 28 11:17:47.561462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561434    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls\") pod \"72a13896-e1c7-4253-8381-db3fee1846f1\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") "
Jul 28 11:17:47.561462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561454    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert\") pod \"72a13896-e1c7-4253-8381-db3fee1846f1\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") "
Jul 28 11:17:47.561802 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561481    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md95p\" (UniqueName: \"kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p\") pod \"72a13896-e1c7-4253-8381-db3fee1846f1\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") "
Jul 28 11:17:47.561802 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561531    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config\") pod \"72a13896-e1c7-4253-8381-db3fee1846f1\" (UID: \"72a13896-e1c7-4253-8381-db3fee1846f1\") "
Jul 28 11:17:47.561954 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561926    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72a13896-e1c7-4253-8381-db3fee1846f1" (UID: "72a13896-e1c7-4253-8381-db3fee1846f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:17:47.562017 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.561972    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72a13896-e1c7-4253-8381-db3fee1846f1" (UID: "72a13896-e1c7-4253-8381-db3fee1846f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:17:47.563574 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.563550    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72a13896-e1c7-4253-8381-db3fee1846f1" (UID: "72a13896-e1c7-4253-8381-db3fee1846f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:17:47.563675 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.563604    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p" (OuterVolumeSpecName: "kube-api-access-md95p") pod "72a13896-e1c7-4253-8381-db3fee1846f1" (UID: "72a13896-e1c7-4253-8381-db3fee1846f1"). InnerVolumeSpecName "kube-api-access-md95p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:17:47.589316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.589272    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72a13896-e1c7-4253-8381-db3fee1846f1" (UID: "72a13896-e1c7-4253-8381-db3fee1846f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:47.662866 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.662832    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72a13896-e1c7-4253-8381-db3fee1846f1-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:47.662866 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.662862    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:47.663032 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.662873    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md95p\" (UniqueName: \"kubernetes.io/projected/72a13896-e1c7-4253-8381-db3fee1846f1-kube-api-access-md95p\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:47.663032 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.662884    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72a13896-e1c7-4253-8381-db3fee1846f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:47.663032 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:47.662893    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72a13896-e1c7-4253-8381-db3fee1846f1-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:17:48.238198 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.238163    2569 generic.go:356] "Generic (PLEG): container finished" podID="72a13896-e1c7-4253-8381-db3fee1846f1" containerID="59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53" exitCode=0
Jul 28 11:17:48.238376 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.238219    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"
Jul 28 11:17:48.238376 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.238244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerDied","Data":"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"}
Jul 28 11:17:48.238376 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.238302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn" event={"ID":"72a13896-e1c7-4253-8381-db3fee1846f1","Type":"ContainerDied","Data":"0d8077698f3a60c65dc4b9775b67d308317dc03b3a27462d59f13fca890b12cf"}
Jul 28 11:17:48.238376 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.238330    2569 scope.go:117] "RemoveContainer" containerID="75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"
Jul 28 11:17:48.246845 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.246827    2569 scope.go:117] "RemoveContainer" containerID="59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"
Jul 28 11:17:48.254025 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.254008    2569 scope.go:117] "RemoveContainer" containerID="7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"
Jul 28 11:17:48.260372 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.260345    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:48.262491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.262474    2569 scope.go:117] "RemoveContainer" containerID="75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"
Jul 28 11:17:48.262872 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:48.262852    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb\": container with ID starting with 75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb not found: ID does not exist" containerID="75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"
Jul 28 11:17:48.262945 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.262880    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb"} err="failed to get container status \"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb\": rpc error: code = NotFound desc = could not find container \"75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb\": container with ID starting with 75bbb391234f9dd7e2b887255511481f751bbe9535b030d59cba7194527ba6bb not found: ID does not exist"
Jul 28 11:17:48.262945 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.262899    2569 scope.go:117] "RemoveContainer" containerID="59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"
Jul 28 11:17:48.263177 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:48.263159    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53\": container with ID starting with 59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53 not found: ID does not exist" containerID="59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"
Jul 28 11:17:48.263220 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.263183    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53"} err="failed to get container status \"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53\": rpc error: code = NotFound desc = could not find container \"59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53\": container with ID starting with 59f3bada810e12f2262e43971e98a02d35b363d2a22fb7aa36b81beb211f5c53 not found: ID does not exist"
Jul 28 11:17:48.263220 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.263201    2569 scope.go:117] "RemoveContainer" containerID="7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"
Jul 28 11:17:48.263475 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:48.263458    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441\": container with ID starting with 7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441 not found: ID does not exist" containerID="7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"
Jul 28 11:17:48.263533 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.263478    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441"} err="failed to get container status \"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441\": rpc error: code = NotFound desc = could not find container \"7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441\": container with ID starting with 7809a84b0ada5b4a2cfa931b743732a6c72cc548a152bfcb80b53efc7bb6b441 not found: ID does not exist"
Jul 28 11:17:48.263573 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.263546    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6c6468c69c-bzxdn"]
Jul 28 11:17:48.567284 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:48.567190    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" path="/var/lib/kubelet/pods/72a13896-e1c7-4253-8381-db3fee1846f1/volumes"
Jul 28 11:17:51.809536 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809503    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809845    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809857    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809873    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809879    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809884    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="storage-initializer"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809893    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="storage-initializer"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809907    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809913    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809920    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="storage-initializer"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809926    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="storage-initializer"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809934    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kube-rbac-proxy"
Jul 28 11:17:51.809943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809939    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kube-rbac-proxy"
Jul 28 11:17:51.810314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.809992    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kserve-container"
Jul 28 11:17:51.810314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.810000    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kserve-container"
Jul 28 11:17:51.810314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.810009    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c08fbe8-2758-496c-8373-45ecdb78b813" containerName="kube-rbac-proxy"
Jul 28 11:17:51.810314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.810014    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a13896-e1c7-4253-8381-db3fee1846f1" containerName="kube-rbac-proxy"
Jul 28 11:17:51.814669 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.814649    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:51.817500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:17:51.817500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817455    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 11:17:51.817500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817464    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:17:51.817500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817447    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:17:51.817832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817463    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 11:17:51.817832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.817477    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:17:51.818354 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.818340    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:17:51.824782 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.824752    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:17:51.899664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.899624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwzn\" (UniqueName: \"kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:51.899664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.899672    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:51.899907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.899713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:51.899907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.899789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:51.899907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:51.899843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000656 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.000620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.000707    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwzn\" (UniqueName: \"kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.000746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000904 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.000846    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000954 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.000900    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.000992 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:52.000974    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 11:17:52.001067 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:17:52.001052    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls podName:6af74700-3a72-4602-8f16-1c72b0157da5 nodeName:}" failed. No retries permitted until 2026-07-28 11:17:52.501031229 +0000 UTC m=+4230.421735288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 11:17:52.001276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.001252    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.001429 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.001411    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.001476 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.001442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.009403 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.009378    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwzn\" (UniqueName: \"kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.505305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.505248    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.507661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.507641    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.725240 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.725211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:52.856911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:52.856887    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:17:52.859649 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:17:52.859622    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af74700_3a72_4602_8f16_1c72b0157da5.slice/crio-799c7333a7f1a64444df7ee9876a9d51afce7aa681c691d2d671df9e558bcfe7 WatchSource:0}: Error finding container 799c7333a7f1a64444df7ee9876a9d51afce7aa681c691d2d671df9e558bcfe7: Status 404 returned error can't find the container with id 799c7333a7f1a64444df7ee9876a9d51afce7aa681c691d2d671df9e558bcfe7
Jul 28 11:17:53.260348 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:53.260310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerStarted","Data":"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"}
Jul 28 11:17:53.260546 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:53.260366    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerStarted","Data":"799c7333a7f1a64444df7ee9876a9d51afce7aa681c691d2d671df9e558bcfe7"}
Jul 28 11:17:57.277654 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:57.277622    2569 generic.go:356] "Generic (PLEG): container finished" podID="6af74700-3a72-4602-8f16-1c72b0157da5" containerID="13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e" exitCode=0
Jul 28 11:17:57.278213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:57.277710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerDied","Data":"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"}
Jul 28 11:17:58.283350 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:58.283317    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerStarted","Data":"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"}
Jul 28 11:17:58.283828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:58.283379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerStarted","Data":"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"}
Jul 28 11:17:58.283828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:58.283709    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:58.283828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:58.283764    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:17:58.303614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:17:58.303563    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" podStartSLOduration=3.731497889 podStartE2EDuration="7.303546748s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.572048859s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:17:58.302438159 +0000 UTC m=+4236.223142211" watchObservedRunningTime="2026-07-28 11:17:58.303546748 +0000 UTC m=+4236.224250800"
Jul 28 11:18:04.293453 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:04.293424    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:18:34.297410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:34.297362    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 11:18:44.296574 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:44.296543    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:18:51.823704 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:51.823654    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:18:51.824190 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:51.823986    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kserve-container" containerID="cri-o://6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841" gracePeriod=30
Jul 28 11:18:51.824190 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:51.824057    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kube-rbac-proxy" containerID="cri-o://19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1" gracePeriod=30
Jul 28 11:18:52.504664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:52.504631    2569 generic.go:356] "Generic (PLEG): container finished" podID="6af74700-3a72-4602-8f16-1c72b0157da5" containerID="19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1" exitCode=2
Jul 28 11:18:52.504849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:52.504720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerDied","Data":"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"}
Jul 28 11:18:54.289511 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:54.289471    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Jul 28 11:18:56.970378 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:56.970343    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:18:56.973824 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:56.973806    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:56.976464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:56.976439    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 11:18:56.976603 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:56.976575    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 11:18:56.985451 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:56.985427    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:18:57.136316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.136276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.136316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.136319    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.136538 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.136335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.136538 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.136390    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554qb\" (UniqueName: \"kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.136538 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.136406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.237794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.237704    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554qb\" (UniqueName: \"kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.237794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.237745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238019 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.237822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238019 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.237853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238019 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.237876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238319 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.238296    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.238635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.238745 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.238635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.240254 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.240236    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.247123 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.247099    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-554qb\" (UniqueName: \"kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb\") pod \"isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.284128 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.284101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:18:57.413014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.412963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:18:57.415653 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:18:57.415627    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e41ab29_2096_43c0_9253_0ee7a008616b.slice/crio-ccb9e214aa1b4f31f6417edc9a36d8dbdcf6acc3a6ebe91cb7a5437e9c1170a0 WatchSource:0}: Error finding container ccb9e214aa1b4f31f6417edc9a36d8dbdcf6acc3a6ebe91cb7a5437e9c1170a0: Status 404 returned error can't find the container with id ccb9e214aa1b4f31f6417edc9a36d8dbdcf6acc3a6ebe91cb7a5437e9c1170a0
Jul 28 11:18:57.525381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.525285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerStarted","Data":"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"}
Jul 28 11:18:57.525381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:57.525344    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerStarted","Data":"ccb9e214aa1b4f31f6417edc9a36d8dbdcf6acc3a6ebe91cb7a5437e9c1170a0"}
Jul 28 11:18:59.163779 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.163751    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:18:59.255723 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.255608    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xwzn\" (UniqueName: \"kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn\") pod \"6af74700-3a72-4602-8f16-1c72b0157da5\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") "
Jul 28 11:18:59.255723 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.255644    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert\") pod \"6af74700-3a72-4602-8f16-1c72b0157da5\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") "
Jul 28 11:18:59.255723 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.255675    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location\") pod \"6af74700-3a72-4602-8f16-1c72b0157da5\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") "
Jul 28 11:18:59.256018 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.255731    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") pod \"6af74700-3a72-4602-8f16-1c72b0157da5\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") "
Jul 28 11:18:59.256018 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.255826    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config\") pod \"6af74700-3a72-4602-8f16-1c72b0157da5\" (UID: \"6af74700-3a72-4602-8f16-1c72b0157da5\") "
Jul 28 11:18:59.256121 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.256028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6af74700-3a72-4602-8f16-1c72b0157da5" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:18:59.256121 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.256063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6af74700-3a72-4602-8f16-1c72b0157da5" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:18:59.256228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.256210    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6af74700-3a72-4602-8f16-1c72b0157da5" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:18:59.257796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.257777    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6af74700-3a72-4602-8f16-1c72b0157da5" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:18:59.257913 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.257894    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn" (OuterVolumeSpecName: "kube-api-access-8xwzn") pod "6af74700-3a72-4602-8f16-1c72b0157da5" (UID: "6af74700-3a72-4602-8f16-1c72b0157da5"). InnerVolumeSpecName "kube-api-access-8xwzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:18:59.356559 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.356527    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwzn\" (UniqueName: \"kubernetes.io/projected/6af74700-3a72-4602-8f16-1c72b0157da5-kube-api-access-8xwzn\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:18:59.356559 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.356553    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:18:59.356559 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.356564    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6af74700-3a72-4602-8f16-1c72b0157da5-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:18:59.356812 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.356574    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6af74700-3a72-4602-8f16-1c72b0157da5-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:18:59.356812 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.356582    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6af74700-3a72-4602-8f16-1c72b0157da5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:18:59.535149 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.535060    2569 generic.go:356] "Generic (PLEG): container finished" podID="6af74700-3a72-4602-8f16-1c72b0157da5" containerID="6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841" exitCode=0
Jul 28 11:18:59.535149 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.535116    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"
Jul 28 11:18:59.535149 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.535135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerDied","Data":"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"}
Jul 28 11:18:59.535386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.535184    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8" event={"ID":"6af74700-3a72-4602-8f16-1c72b0157da5","Type":"ContainerDied","Data":"799c7333a7f1a64444df7ee9876a9d51afce7aa681c691d2d671df9e558bcfe7"}
Jul 28 11:18:59.535386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.535209    2569 scope.go:117] "RemoveContainer" containerID="19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"
Jul 28 11:18:59.543380 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.543362    2569 scope.go:117] "RemoveContainer" containerID="6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"
Jul 28 11:18:59.550748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.550730    2569 scope.go:117] "RemoveContainer" containerID="13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"
Jul 28 11:18:59.557009 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.556986    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:18:59.557846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.557828    2569 scope.go:117] "RemoveContainer" containerID="19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"
Jul 28 11:18:59.558095 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:18:59.558075    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1\": container with ID starting with 19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1 not found: ID does not exist" containerID="19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"
Jul 28 11:18:59.558144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.558104    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1"} err="failed to get container status \"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1\": rpc error: code = NotFound desc = could not find container \"19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1\": container with ID starting with 19830a1bb8556e3b0774aef5b1998a6f63aae021df9b7e8a5614d9c48e86cca1 not found: ID does not exist"
Jul 28 11:18:59.558144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.558122    2569 scope.go:117] "RemoveContainer" containerID="6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"
Jul 28 11:18:59.558327 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:18:59.558309    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841\": container with ID starting with 6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841 not found: ID does not exist" containerID="6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"
Jul 28 11:18:59.558395 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.558335    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841"} err="failed to get container status \"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841\": rpc error: code = NotFound desc = could not find container \"6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841\": container with ID starting with 6661ce59cac25ae8dd4c2da0f568fd1f19ed435a2e0a4ce92677aa9fde408841 not found: ID does not exist"
Jul 28 11:18:59.558395 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.558356    2569 scope.go:117] "RemoveContainer" containerID="13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"
Jul 28 11:18:59.558629 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:18:59.558599    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e\": container with ID starting with 13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e not found: ID does not exist" containerID="13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"
Jul 28 11:18:59.558702 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.558627    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e"} err="failed to get container status \"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e\": rpc error: code = NotFound desc = could not find container \"13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e\": container with ID starting with 13e3e6d25223f072b7ec2667fa27492ed0bac7ea49cae0f53e84538a611fc67e not found: ID does not exist"
Jul 28 11:18:59.560489 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:18:59.560471    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-kjcq8"]
Jul 28 11:19:00.565952 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:00.565912    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" path="/var/lib/kubelet/pods/6af74700-3a72-4602-8f16-1c72b0157da5/volumes"
Jul 28 11:19:01.545152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:01.545069    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerID="bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353" exitCode=0
Jul 28 11:19:01.545152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:01.545108    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerDied","Data":"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"}
Jul 28 11:19:02.551318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:02.551285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerStarted","Data":"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"}
Jul 28 11:19:02.551729 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:02.551351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerStarted","Data":"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"}
Jul 28 11:19:02.551729 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:02.551547    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:19:02.573288 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:02.573146    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podStartSLOduration=2.862154328 podStartE2EDuration="6.573126522s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.710972194s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:19:02.569843256 +0000 UTC m=+4300.490547309" watchObservedRunningTime="2026-07-28 11:19:02.573126522 +0000 UTC m=+4300.493830574"
Jul 28 11:19:03.555887 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:03.555853    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:19:03.557089 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:03.557046    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:04.560385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:04.560343    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:09.564881 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:09.564849    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:19:09.565354 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:09.565325    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:19.565863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:19.565783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:29.565548 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:29.565507    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:39.566204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:39.566159    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:49.565826 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:49.565789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:19:59.565966 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:19:59.565927    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:20:09.565779 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:09.565742    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:20:17.561841 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:17.561810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:20:27.224381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.224339    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:20:27.224921 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.224635    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" containerID="cri-o://045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f" gracePeriod=30
Jul 28 11:20:27.224921 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.224738    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kube-rbac-proxy" containerID="cri-o://3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125" gracePeriod=30
Jul 28 11:20:27.562528 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.562429    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Jul 28 11:20:27.893296 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.893204    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerID="3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125" exitCode=2
Jul 28 11:20:27.893296 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:27.893272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerDied","Data":"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"}
Jul 28 11:20:29.560893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:29.560847    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Jul 28 11:20:32.169211 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.169186    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:20:32.237986 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.237947    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location\") pod \"3e41ab29-2096-43c0-9253-0ee7a008616b\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") "
Jul 28 11:20:32.238176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238023    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls\") pod \"3e41ab29-2096-43c0-9253-0ee7a008616b\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") "
Jul 28 11:20:32.238176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert\") pod \"3e41ab29-2096-43c0-9253-0ee7a008616b\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") "
Jul 28 11:20:32.238176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-554qb\" (UniqueName: \"kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb\") pod \"3e41ab29-2096-43c0-9253-0ee7a008616b\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") "
Jul 28 11:20:32.238176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238137    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config\") pod \"3e41ab29-2096-43c0-9253-0ee7a008616b\" (UID: \"3e41ab29-2096-43c0-9253-0ee7a008616b\") "
Jul 28 11:20:32.238420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238282    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e41ab29-2096-43c0-9253-0ee7a008616b" (UID: "3e41ab29-2096-43c0-9253-0ee7a008616b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:20:32.238534 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238487    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e41ab29-2096-43c0-9253-0ee7a008616b" (UID: "3e41ab29-2096-43c0-9253-0ee7a008616b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:20:32.238534 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.238520    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e41ab29-2096-43c0-9253-0ee7a008616b" (UID: "3e41ab29-2096-43c0-9253-0ee7a008616b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:20:32.240175 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.240156    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e41ab29-2096-43c0-9253-0ee7a008616b" (UID: "3e41ab29-2096-43c0-9253-0ee7a008616b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:20:32.240263 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.240244    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb" (OuterVolumeSpecName: "kube-api-access-554qb") pod "3e41ab29-2096-43c0-9253-0ee7a008616b" (UID: "3e41ab29-2096-43c0-9253-0ee7a008616b"). InnerVolumeSpecName "kube-api-access-554qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:20:32.339719 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.339665    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554qb\" (UniqueName: \"kubernetes.io/projected/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-api-access-554qb\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:20:32.339719 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.339711    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:20:32.339719 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.339721    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e41ab29-2096-43c0-9253-0ee7a008616b-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:20:32.339975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.339731    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e41ab29-2096-43c0-9253-0ee7a008616b-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:20:32.339975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.339741    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e41ab29-2096-43c0-9253-0ee7a008616b-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:20:32.379795 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.379759    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:20:32.380102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380090    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380104    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380116    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380123    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380131    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380137    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="storage-initializer"
Jul 28 11:20:32.380154 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380151    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="storage-initializer"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380163    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="storage-initializer"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380168    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="storage-initializer"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kserve-container"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380181    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kserve-container"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380228    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380236    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kube-rbac-proxy"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380242    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerName="kserve-container"
Jul 28 11:20:32.380381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.380249    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6af74700-3a72-4602-8f16-1c72b0157da5" containerName="kserve-container"
Jul 28 11:20:32.383457 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.383440    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.385988 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.385969    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 11:20:32.386272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.386252    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 11:20:32.394232 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.394206    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:20:32.541753 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.541661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.541753 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.541734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.541973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.541794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fdd\" (UniqueName: \"kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.541973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.541875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.541973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.541903    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643235 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643235 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643519 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643519 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643519 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fdd\" (UniqueName: \"kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.643722 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.643675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.644039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.644018    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.644144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.644127    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.645850 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.645832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.651132 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.651114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fdd\" (UniqueName: \"kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd\") pod \"isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.694194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.694161    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:32.825636 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.825610    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:20:32.827907 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:20:32.827885    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e5a4d9_6988_436e_b7d9_ecde11da8359.slice/crio-80e97aec40811a88f04f0f5f09ba1d41faf4f09aaef15f10a662c5cfd38e9cc0 WatchSource:0}: Error finding container 80e97aec40811a88f04f0f5f09ba1d41faf4f09aaef15f10a662c5cfd38e9cc0: Status 404 returned error can't find the container with id 80e97aec40811a88f04f0f5f09ba1d41faf4f09aaef15f10a662c5cfd38e9cc0
Jul 28 11:20:32.915749 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.915710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerStarted","Data":"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"}
Jul 28 11:20:32.915939 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.915786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerStarted","Data":"80e97aec40811a88f04f0f5f09ba1d41faf4f09aaef15f10a662c5cfd38e9cc0"}
Jul 28 11:20:32.917476 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.917447    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e41ab29-2096-43c0-9253-0ee7a008616b" containerID="045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f" exitCode=0
Jul 28 11:20:32.917620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.917504    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"
Jul 28 11:20:32.917620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.917516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerDied","Data":"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"}
Jul 28 11:20:32.917620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.917565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl" event={"ID":"3e41ab29-2096-43c0-9253-0ee7a008616b","Type":"ContainerDied","Data":"ccb9e214aa1b4f31f6417edc9a36d8dbdcf6acc3a6ebe91cb7a5437e9c1170a0"}
Jul 28 11:20:32.917620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.917593    2569 scope.go:117] "RemoveContainer" containerID="3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"
Jul 28 11:20:32.926327 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.926309    2569 scope.go:117] "RemoveContainer" containerID="045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"
Jul 28 11:20:32.934609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.934582    2569 scope.go:117] "RemoveContainer" containerID="bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"
Jul 28 11:20:32.942553 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.942534    2569 scope.go:117] "RemoveContainer" containerID="3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"
Jul 28 11:20:32.942839 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:20:32.942822    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125\": container with ID starting with 3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125 not found: ID does not exist" containerID="3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"
Jul 28 11:20:32.942899 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.942846    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125"} err="failed to get container status \"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125\": rpc error: code = NotFound desc = could not find container \"3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125\": container with ID starting with 3d0e18cdd1635ffb7756c651cf6445d100572766f0052c7b8ff39776a871a125 not found: ID does not exist"
Jul 28 11:20:32.942899 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.942863    2569 scope.go:117] "RemoveContainer" containerID="045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"
Jul 28 11:20:32.943146 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:20:32.943126    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f\": container with ID starting with 045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f not found: ID does not exist" containerID="045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"
Jul 28 11:20:32.943216 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.943153    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f"} err="failed to get container status \"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f\": rpc error: code = NotFound desc = could not find container \"045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f\": container with ID starting with 045d1ef1e1642dfe97d1b0b04ca29140fd578792b7c638c2a69e212ceb3d954f not found: ID does not exist"
Jul 28 11:20:32.943216 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.943171    2569 scope.go:117] "RemoveContainer" containerID="bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"
Jul 28 11:20:32.943495 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:20:32.943467    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353\": container with ID starting with bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353 not found: ID does not exist" containerID="bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"
Jul 28 11:20:32.943567 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.943504    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353"} err="failed to get container status \"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353\": rpc error: code = NotFound desc = could not find container \"bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353\": container with ID starting with bd69041e9a4032a310da58ef10aac1b8b77f66085c3cb4ec94ea2e1eea117353 not found: ID does not exist"
Jul 28 11:20:32.950860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.950835    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:20:32.955170 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:32.955148    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6dbb4b5c65-lvrnl"]
Jul 28 11:20:34.566067 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:34.566027    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e41ab29-2096-43c0-9253-0ee7a008616b" path="/var/lib/kubelet/pods/3e41ab29-2096-43c0-9253-0ee7a008616b/volumes"
Jul 28 11:20:36.937760 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:36.937652    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerID="7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9" exitCode=0
Jul 28 11:20:36.937760 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:36.937724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerDied","Data":"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"}
Jul 28 11:20:37.943551 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:37.943514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerStarted","Data":"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"}
Jul 28 11:20:37.943966 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:37.943575    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerStarted","Data":"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"}
Jul 28 11:20:37.943966 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:37.943747    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:37.966318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:37.966270    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podStartSLOduration=2.175103502 podStartE2EDuration="5.966254402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7911509s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:20:37.965935787 +0000 UTC m=+4395.886639838" watchObservedRunningTime="2026-07-28 11:20:37.966254402 +0000 UTC m=+4395.886958455"
Jul 28 11:20:38.947973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:38.947933    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:38.949224 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:38.949200    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:20:39.952049 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:39.952009    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:20:44.956860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:44.956790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:20:44.957393 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:44.957369    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:20:54.957701 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:20:54.957650    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:04.957733 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:04.957673    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:14.957652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:14.957609    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:24.957525 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:24.957483    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:34.957710 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:34.957646    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:44.957611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:44.957566    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:48.566889 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:48.566854    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 11:21:58.567735 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:21:58.567705    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:22:02.585053 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:02.585019    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:22:02.585544 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:02.585265    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container" containerID="cri-o://a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624" gracePeriod=30
Jul 28 11:22:02.585544 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:02.585296    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kube-rbac-proxy" containerID="cri-o://62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808" gracePeriod=30
Jul 28 11:22:03.292482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:03.292447    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerID="62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808" exitCode=2
Jul 28 11:22:03.292658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:03.292521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerDied","Data":"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"}
Jul 28 11:22:04.952785 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:04.952744    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Jul 28 11:22:07.636635 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.636608    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:22:07.733023 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.732984    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location\") pod \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") "
Jul 28 11:22:07.733210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733090    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config\") pod \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") "
Jul 28 11:22:07.733210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733119    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert\") pod \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") "
Jul 28 11:22:07.733210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6fdd\" (UniqueName: \"kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd\") pod \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") "
Jul 28 11:22:07.733210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733185    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls\") pod \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\" (UID: \"e3e5a4d9-6988-436e-b7d9-ecde11da8359\") "
Jul 28 11:22:07.733418 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733319    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3e5a4d9-6988-436e-b7d9-ecde11da8359" (UID: "e3e5a4d9-6988-436e-b7d9-ecde11da8359"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:22:07.733474 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733425    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3e5a4d9-6988-436e-b7d9-ecde11da8359" (UID: "e3e5a4d9-6988-436e-b7d9-ecde11da8359"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:22:07.733518 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733476    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3e5a4d9-6988-436e-b7d9-ecde11da8359" (UID: "e3e5a4d9-6988-436e-b7d9-ecde11da8359"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:22:07.733587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733567    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:22:07.733625 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.733596    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:22:07.735228 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.735193    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd" (OuterVolumeSpecName: "kube-api-access-p6fdd") pod "e3e5a4d9-6988-436e-b7d9-ecde11da8359" (UID: "e3e5a4d9-6988-436e-b7d9-ecde11da8359"). InnerVolumeSpecName "kube-api-access-p6fdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:22:07.735336 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.735268    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3e5a4d9-6988-436e-b7d9-ecde11da8359" (UID: "e3e5a4d9-6988-436e-b7d9-ecde11da8359"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:22:07.746020 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.745994    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:22:07.746310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746299    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kube-rbac-proxy"
Jul 28 11:22:07.746353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kube-rbac-proxy"
Jul 28 11:22:07.746353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746325    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="storage-initializer"
Jul 28 11:22:07.746353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746331    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="storage-initializer"
Jul 28 11:22:07.746353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container"
Jul 28 11:22:07.746353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746354    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container"
Jul 28 11:22:07.746508 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746407    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kserve-container"
Jul 28 11:22:07.746508 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.746417    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerName="kube-rbac-proxy"
Jul 28 11:22:07.749597 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.749583    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.752240 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.752222    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 11:22:07.752334 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.752250    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 11:22:07.759000 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.758971    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:22:07.834620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.834836 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.834836 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.834836 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwj8\" (UniqueName: \"kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.834941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834841    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.834941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834916    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3e5a4d9-6988-436e-b7d9-ecde11da8359-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:22:07.834941 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834930    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fdd\" (UniqueName: \"kubernetes.io/projected/e3e5a4d9-6988-436e-b7d9-ecde11da8359-kube-api-access-p6fdd\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:22:07.835039 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.834943    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3e5a4d9-6988-436e-b7d9-ecde11da8359-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:22:07.936213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.936213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936225    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.936497 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.936497 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwj8\" (UniqueName: \"kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.936497 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.936710 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.937010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.936978    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.937010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.937003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.938805 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.938787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:07.944507 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:07.944489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwj8\" (UniqueName: \"kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8\") pod \"isvc-tensorflow-predictor-547dfc5669-cf4n6\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:08.060106 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.060008    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:08.190998 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.190972    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:22:08.193347 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:22:08.193317    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536c347b_3895_4891_9be3_c01b303c7365.slice/crio-1d785d21d695d6af77807399e9319cf2074b41cd3b4ac50ace6745a04c2c9ec7 WatchSource:0}: Error finding container 1d785d21d695d6af77807399e9319cf2074b41cd3b4ac50ace6745a04c2c9ec7: Status 404 returned error can't find the container with id 1d785d21d695d6af77807399e9319cf2074b41cd3b4ac50ace6745a04c2c9ec7
Jul 28 11:22:08.316229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.316135    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" containerID="a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624" exitCode=0
Jul 28 11:22:08.316229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.316201    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"
Jul 28 11:22:08.316229 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.316212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerDied","Data":"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"}
Jul 28 11:22:08.316527 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.316289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb" event={"ID":"e3e5a4d9-6988-436e-b7d9-ecde11da8359","Type":"ContainerDied","Data":"80e97aec40811a88f04f0f5f09ba1d41faf4f09aaef15f10a662c5cfd38e9cc0"}
Jul 28 11:22:08.316527 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.316329    2569 scope.go:117] "RemoveContainer" containerID="62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"
Jul 28 11:22:08.318026 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.317988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerStarted","Data":"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"}
Jul 28 11:22:08.318167 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.318050    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerStarted","Data":"1d785d21d695d6af77807399e9319cf2074b41cd3b4ac50ace6745a04c2c9ec7"}
Jul 28 11:22:08.324940 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.324921    2569 scope.go:117] "RemoveContainer" containerID="a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"
Jul 28 11:22:08.333953 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.333937    2569 scope.go:117] "RemoveContainer" containerID="7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"
Jul 28 11:22:08.341618 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.341584    2569 scope.go:117] "RemoveContainer" containerID="62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"
Jul 28 11:22:08.342015 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:22:08.341985    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808\": container with ID starting with 62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808 not found: ID does not exist" containerID="62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"
Jul 28 11:22:08.342138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.342027    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808"} err="failed to get container status \"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808\": rpc error: code = NotFound desc = could not find container \"62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808\": container with ID starting with 62b6d358427035346c81172c5b8b610bab7b5f83cf3b6e6b621dfe6077701808 not found: ID does not exist"
Jul 28 11:22:08.342138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.342054    2569 scope.go:117] "RemoveContainer" containerID="a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"
Jul 28 11:22:08.342361 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:22:08.342338    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624\": container with ID starting with a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624 not found: ID does not exist" containerID="a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"
Jul 28 11:22:08.342408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.342366    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624"} err="failed to get container status \"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624\": rpc error: code = NotFound desc = could not find container \"a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624\": container with ID starting with a08cb7a4d96c2711842d8b5159a7d171743704952f78e625515eca712b667624 not found: ID does not exist"
Jul 28 11:22:08.342408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.342382    2569 scope.go:117] "RemoveContainer" containerID="7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"
Jul 28 11:22:08.342657 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:22:08.342634    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9\": container with ID starting with 7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9 not found: ID does not exist" containerID="7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"
Jul 28 11:22:08.342748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.342667    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9"} err="failed to get container status \"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9\": rpc error: code = NotFound desc = could not find container \"7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9\": container with ID starting with 7fbee9d5fe1237c52f56d01fdd70c71087027a5e867b326c25b12bcc27e210d9 not found: ID does not exist"
Jul 28 11:22:08.355660 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.355627    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:22:08.361159 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.361127    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-56cbd9f7d4-qf9cb"]
Jul 28 11:22:08.566464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:08.566395    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e5a4d9-6988-436e-b7d9-ecde11da8359" path="/var/lib/kubelet/pods/e3e5a4d9-6988-436e-b7d9-ecde11da8359/volumes"
Jul 28 11:22:13.346173 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:13.346137    2569 generic.go:356] "Generic (PLEG): container finished" podID="536c347b-3895-4891-9be3-c01b303c7365" containerID="a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722" exitCode=0
Jul 28 11:22:13.346670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:13.346215    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerDied","Data":"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"}
Jul 28 11:22:13.347569 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:13.347554    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:22:17.368377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:17.368336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerStarted","Data":"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"}
Jul 28 11:22:17.368832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:17.368452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerStarted","Data":"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"}
Jul 28 11:22:17.368832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:17.368657    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:17.390620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:17.390569    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podStartSLOduration=1.994479855 podStartE2EDuration="10.390552816s" totalImagesPullingTime="3.851705266s" totalInitContainerRuntime="4.544367695s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:22:17.388741594 +0000 UTC m=+4495.309445643" watchObservedRunningTime="2026-07-28 11:22:17.390552816 +0000 UTC m=+4495.311256867"
Jul 28 11:22:18.372774 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:18.372743    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:18.373960 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:18.373931    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Jul 28 11:22:19.377591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:19.377555    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Jul 28 11:22:22.780986 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:22.780961    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:22:22.794790 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:22.794759    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:22:24.382372 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:24.382344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:24.382907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:24.382883    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Jul 28 11:22:34.383062 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:34.383034    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:48.647800 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:48.647766    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:22:48.648390 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:48.648062    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container" containerID="cri-o://265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66" gracePeriod=30
Jul 28 11:22:48.648390 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:48.648162    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" containerID="cri-o://df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705" gracePeriod=30
Jul 28 11:22:49.377804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:49.377765    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:22:49.498660 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:49.498626    2569 generic.go:356] "Generic (PLEG): container finished" podID="536c347b-3895-4891-9be3-c01b303c7365" containerID="df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705" exitCode=2
Jul 28 11:22:49.498848 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:49.498711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerDied","Data":"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"}
Jul 28 11:22:53.824029 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.823991    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:22:53.827187 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.827171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:53.829635 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.829614    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 11:22:53.829635 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.829614    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 11:22:53.839751 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.839727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:22:53.926442 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.926406    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:53.926664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.926451    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:53.926664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.926499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:53.926664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.926525    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:53.926664 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:53.926606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjvm\" (UniqueName: \"kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.027875 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.027834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.027875 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.027883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.027913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.027939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.028002    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjvm\" (UniqueName: \"kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028356 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.028334    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.028597    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.028665 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.028624    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.030359 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.030341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.036008 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.035984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjvm\" (UniqueName: \"kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.137102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.137007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:22:54.268368 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.268345    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:22:54.270194 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:22:54.270164    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7df6eb1_0ff4_4b50_946b_1588ff35f319.slice/crio-9e83f1912ecf86c50a7b420b6eab8a439690b5fd199c87d34517fe90b512ead5 WatchSource:0}: Error finding container 9e83f1912ecf86c50a7b420b6eab8a439690b5fd199c87d34517fe90b512ead5: Status 404 returned error can't find the container with id 9e83f1912ecf86c50a7b420b6eab8a439690b5fd199c87d34517fe90b512ead5
Jul 28 11:22:54.378602 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.378558    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:22:54.526550 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.526507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerStarted","Data":"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"}
Jul 28 11:22:54.526745 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:54.526598    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerStarted","Data":"9e83f1912ecf86c50a7b420b6eab8a439690b5fd199c87d34517fe90b512ead5"}
Jul 28 11:22:59.378364 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:59.378321    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:22:59.378856 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:59.378442    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:22:59.550679 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:59.550643    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerID="826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd" exitCode=0
Jul 28 11:22:59.550864 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:22:59.550721    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerDied","Data":"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"}
Jul 28 11:23:00.557103 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:00.557067    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerStarted","Data":"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"}
Jul 28 11:23:00.557598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:00.557139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerStarted","Data":"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"}
Jul 28 11:23:00.557598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:00.557351    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:23:00.577449 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:00.577393    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podStartSLOduration=3.366429617 podStartE2EDuration="7.577376371s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.210946754s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:23:00.574998872 +0000 UTC m=+4538.495702925" watchObservedRunningTime="2026-07-28 11:23:00.577376371 +0000 UTC m=+4538.498080422"
Jul 28 11:23:01.561632 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:01.561602    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:23:01.562706 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:01.562665    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Jul 28 11:23:02.565581 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:02.565540    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Jul 28 11:23:04.378587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:04.378544    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:23:07.570200 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:07.570172    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:23:07.570783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:07.570754    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Jul 28 11:23:09.378466 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:09.378424    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:23:14.378549 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:14.378510    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Jul 28 11:23:17.571599 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:17.571571    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:23:19.285237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.285212    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:23:19.457753 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.457649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svwj8\" (UniqueName: \"kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8\") pod \"536c347b-3895-4891-9be3-c01b303c7365\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") "
Jul 28 11:23:19.457753 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.457738    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config\") pod \"536c347b-3895-4891-9be3-c01b303c7365\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") "
Jul 28 11:23:19.457994 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.457838    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert\") pod \"536c347b-3895-4891-9be3-c01b303c7365\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") "
Jul 28 11:23:19.457994 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.457902    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls\") pod \"536c347b-3895-4891-9be3-c01b303c7365\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") "
Jul 28 11:23:19.457994 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.457972    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location\") pod \"536c347b-3895-4891-9be3-c01b303c7365\" (UID: \"536c347b-3895-4891-9be3-c01b303c7365\") "
Jul 28 11:23:19.458168 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.458089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "536c347b-3895-4891-9be3-c01b303c7365" (UID: "536c347b-3895-4891-9be3-c01b303c7365"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:23:19.458305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.458239    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "536c347b-3895-4891-9be3-c01b303c7365" (UID: "536c347b-3895-4891-9be3-c01b303c7365"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:23:19.458305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.458290    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:23:19.459942 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.459913    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "536c347b-3895-4891-9be3-c01b303c7365" (UID: "536c347b-3895-4891-9be3-c01b303c7365"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:23:19.460032 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.459978    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8" (OuterVolumeSpecName: "kube-api-access-svwj8") pod "536c347b-3895-4891-9be3-c01b303c7365" (UID: "536c347b-3895-4891-9be3-c01b303c7365"). InnerVolumeSpecName "kube-api-access-svwj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:23:19.469100 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.469073    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "536c347b-3895-4891-9be3-c01b303c7365" (UID: "536c347b-3895-4891-9be3-c01b303c7365"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:19.559566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.559519    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c347b-3895-4891-9be3-c01b303c7365-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:23:19.559566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.559563    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwj8\" (UniqueName: \"kubernetes.io/projected/536c347b-3895-4891-9be3-c01b303c7365-kube-api-access-svwj8\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:23:19.559566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.559573    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c347b-3895-4891-9be3-c01b303c7365-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:23:19.559566 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.559585    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c347b-3895-4891-9be3-c01b303c7365-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:23:19.636104 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.636068    2569 generic.go:356] "Generic (PLEG): container finished" podID="536c347b-3895-4891-9be3-c01b303c7365" containerID="265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66" exitCode=137
Jul 28 11:23:19.636307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.636132    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerDied","Data":"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"}
Jul 28 11:23:19.636307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.636134    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"
Jul 28 11:23:19.636307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.636179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6" event={"ID":"536c347b-3895-4891-9be3-c01b303c7365","Type":"ContainerDied","Data":"1d785d21d695d6af77807399e9319cf2074b41cd3b4ac50ace6745a04c2c9ec7"}
Jul 28 11:23:19.636307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.636204    2569 scope.go:117] "RemoveContainer" containerID="df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"
Jul 28 11:23:19.644418 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.644400    2569 scope.go:117] "RemoveContainer" containerID="265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"
Jul 28 11:23:19.651546 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.651531    2569 scope.go:117] "RemoveContainer" containerID="a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"
Jul 28 11:23:19.658877 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.658857    2569 scope.go:117] "RemoveContainer" containerID="df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"
Jul 28 11:23:19.659187 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:23:19.659161    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705\": container with ID starting with df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705 not found: ID does not exist" containerID="df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"
Jul 28 11:23:19.659271 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.659201    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705"} err="failed to get container status \"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705\": rpc error: code = NotFound desc = could not find container \"df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705\": container with ID starting with df58d3601c838c2e7fd9cebb89e47eca5b9c2a5148f11684787125c28e2a9705 not found: ID does not exist"
Jul 28 11:23:19.659271 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.659224    2569 scope.go:117] "RemoveContainer" containerID="265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"
Jul 28 11:23:19.659502 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:23:19.659480    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66\": container with ID starting with 265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66 not found: ID does not exist" containerID="265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"
Jul 28 11:23:19.659562 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.659510    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66"} err="failed to get container status \"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66\": rpc error: code = NotFound desc = could not find container \"265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66\": container with ID starting with 265590212bdf1b9504044444bf6f5eacb54bb9890ea21eee9502e62bcc6e4a66 not found: ID does not exist"
Jul 28 11:23:19.659562 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.659531    2569 scope.go:117] "RemoveContainer" containerID="a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"
Jul 28 11:23:19.659775 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:23:19.659755    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722\": container with ID starting with a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722 not found: ID does not exist" containerID="a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"
Jul 28 11:23:19.659873 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.659779    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722"} err="failed to get container status \"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722\": rpc error: code = NotFound desc = could not find container \"a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722\": container with ID starting with a25f8dd6c75511f935836e1fc6e05e91e7744493515e1ed70effed5d6acb2722 not found: ID does not exist"
Jul 28 11:23:19.660727 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.660706    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:23:19.664441 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:19.664421    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-cf4n6"]
Jul 28 11:23:20.565817 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:20.565775    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536c347b-3895-4891-9be3-c01b303c7365" path="/var/lib/kubelet/pods/536c347b-3895-4891-9be3-c01b303c7365/volumes"
Jul 28 11:23:34.947411 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:34.947376    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:23:34.947973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:34.947670    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" containerID="cri-o://fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578" gracePeriod=30
Jul 28 11:23:34.947973 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:34.947656    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container" containerID="cri-o://35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267" gracePeriod=30
Jul 28 11:23:35.699432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:35.699398    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerID="fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578" exitCode=2
Jul 28 11:23:35.699609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:35.699450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerDied","Data":"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"}
Jul 28 11:23:37.565828 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:37.565785    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:23:40.137277 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137229    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137535    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="storage-initializer"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137546    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="storage-initializer"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137566    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137571    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137579    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137585    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137634    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kube-rbac-proxy"
Jul 28 11:23:40.137713 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.137641    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="536c347b-3895-4891-9be3-c01b303c7365" containerName="kserve-container"
Jul 28 11:23:40.140771 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.140755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.143212 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.143186    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 11:23:40.143350 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.143187    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 11:23:40.150034 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.150010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:23:40.228758 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.228713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.228758 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.228768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.228980 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.228836    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.228980 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.228873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hd8\" (UniqueName: \"kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.228980 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.228944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330005 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.329966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hd8\" (UniqueName: \"kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330098    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330440 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:23:40.330204    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Jul 28 11:23:40.330440 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:23:40.330292    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls podName:883a7475-e4ff-48f0-9380-e4ec60f5f24e nodeName:}" failed. No retries permitted until 2026-07-28 11:23:40.830271674 +0000 UTC m=+4578.750975703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls") pod "isvc-triton-predictor-5d7db58676-rjs9b" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e") : secret "isvc-triton-predictor-serving-cert" not found
Jul 28 11:23:40.330558 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330711 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.330756 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.330728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.341834 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.341812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hd8\" (UniqueName: \"kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.835431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.835388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:40.837787 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:40.837755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-rjs9b\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:41.051060 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:41.051007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:23:41.186438 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:41.186406    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:23:41.189829 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:23:41.189802    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883a7475_e4ff_48f0_9380_e4ec60f5f24e.slice/crio-52734d3ab145d217b9c8d40bcf5f455542dcc00f91b238e9465bd9578a2a9dcd WatchSource:0}: Error finding container 52734d3ab145d217b9c8d40bcf5f455542dcc00f91b238e9465bd9578a2a9dcd: Status 404 returned error can't find the container with id 52734d3ab145d217b9c8d40bcf5f455542dcc00f91b238e9465bd9578a2a9dcd
Jul 28 11:23:41.723390 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:41.723354    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerStarted","Data":"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"}
Jul 28 11:23:41.723561 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:41.723411    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerStarted","Data":"52734d3ab145d217b9c8d40bcf5f455542dcc00f91b238e9465bd9578a2a9dcd"}
Jul 28 11:23:42.566605 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:42.566560    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:23:45.742884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:45.742851    2569 generic.go:356] "Generic (PLEG): container finished" podID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerID="62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2" exitCode=0
Jul 28 11:23:45.743274 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:45.742923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerDied","Data":"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"}
Jul 28 11:23:47.567147 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:47.566452    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:23:47.567147 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:47.566604    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:23:52.565667 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:52.565627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:23:57.566980 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:23:57.566527    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:24:02.565631 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:02.565594    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Jul 28 11:24:05.647594 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.647564    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:24:05.775097 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.774966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location\") pod \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") "
Jul 28 11:24:05.775097 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.775035    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls\") pod \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") "
Jul 28 11:24:05.775097 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.775068    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config\") pod \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") "
Jul 28 11:24:05.775474 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.775107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert\") pod \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") "
Jul 28 11:24:05.775474 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.775145    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkjvm\" (UniqueName: \"kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm\") pod \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\" (UID: \"f7df6eb1-0ff4-4b50-946b-1588ff35f319\") "
Jul 28 11:24:05.775928 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.775899    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7df6eb1-0ff4-4b50-946b-1588ff35f319" (UID: "f7df6eb1-0ff4-4b50-946b-1588ff35f319"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:05.776080 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.776062    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7df6eb1-0ff4-4b50-946b-1588ff35f319" (UID: "f7df6eb1-0ff4-4b50-946b-1588ff35f319"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:05.778073 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.778046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm" (OuterVolumeSpecName: "kube-api-access-pkjvm") pod "f7df6eb1-0ff4-4b50-946b-1588ff35f319" (UID: "f7df6eb1-0ff4-4b50-946b-1588ff35f319"). InnerVolumeSpecName "kube-api-access-pkjvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:05.779353 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.779313    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7df6eb1-0ff4-4b50-946b-1588ff35f319" (UID: "f7df6eb1-0ff4-4b50-946b-1588ff35f319"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:24:05.782034 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.782004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7df6eb1-0ff4-4b50-946b-1588ff35f319" (UID: "f7df6eb1-0ff4-4b50-946b-1588ff35f319"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:05.876504 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.876470    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:24:05.876504 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.876502    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7df6eb1-0ff4-4b50-946b-1588ff35f319-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:24:05.876504 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.876513    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:24:05.876926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.876524    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7df6eb1-0ff4-4b50-946b-1588ff35f319-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:24:05.876926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.876534    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjvm\" (UniqueName: \"kubernetes.io/projected/f7df6eb1-0ff4-4b50-946b-1588ff35f319-kube-api-access-pkjvm\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:24:05.891094 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.891057    2569 generic.go:356] "Generic (PLEG): container finished" podID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerID="35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267" exitCode=137
Jul 28 11:24:05.891322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.891123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerDied","Data":"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"}
Jul 28 11:24:05.891322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.891145    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"
Jul 28 11:24:05.891322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.891198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb" event={"ID":"f7df6eb1-0ff4-4b50-946b-1588ff35f319","Type":"ContainerDied","Data":"9e83f1912ecf86c50a7b420b6eab8a439690b5fd199c87d34517fe90b512ead5"}
Jul 28 11:24:05.891322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.891239    2569 scope.go:117] "RemoveContainer" containerID="fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"
Jul 28 11:24:05.903410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.903031    2569 scope.go:117] "RemoveContainer" containerID="35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"
Jul 28 11:24:05.913067 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.913038    2569 scope.go:117] "RemoveContainer" containerID="826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"
Jul 28 11:24:05.919343 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.919307    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:24:05.922109 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.922085    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-6sqxb"]
Jul 28 11:24:05.924415 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.924384    2569 scope.go:117] "RemoveContainer" containerID="fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"
Jul 28 11:24:05.924880 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:24:05.924723    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578\": container with ID starting with fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578 not found: ID does not exist" containerID="fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"
Jul 28 11:24:05.924880 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.924759    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578"} err="failed to get container status \"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578\": rpc error: code = NotFound desc = could not find container \"fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578\": container with ID starting with fc7f7e98ca3da9c3e9de378342445644879caee05f3bbab9b4cf3c04fe3f4578 not found: ID does not exist"
Jul 28 11:24:05.924880 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.924786    2569 scope.go:117] "RemoveContainer" containerID="35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"
Jul 28 11:24:05.925171 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:24:05.925144    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267\": container with ID starting with 35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267 not found: ID does not exist" containerID="35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"
Jul 28 11:24:05.925287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.925178    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267"} err="failed to get container status \"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267\": rpc error: code = NotFound desc = could not find container \"35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267\": container with ID starting with 35ee23ffb2d7e68f67081314d5d8bef55c3d6c70ab7fa3ed2c1e6f1456aa4267 not found: ID does not exist"
Jul 28 11:24:05.925287 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.925200    2569 scope.go:117] "RemoveContainer" containerID="826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"
Jul 28 11:24:05.925527 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:24:05.925497    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd\": container with ID starting with 826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd not found: ID does not exist" containerID="826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"
Jul 28 11:24:05.925653 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:05.925533    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd"} err="failed to get container status \"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd\": rpc error: code = NotFound desc = could not find container \"826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd\": container with ID starting with 826b3671b30d61af8064438186521c0c6b3d4cf00cda1a587e6964b265c2c0bd not found: ID does not exist"
Jul 28 11:24:06.569318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:24:06.569148    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" path="/var/lib/kubelet/pods/f7df6eb1-0ff4-4b50-946b-1588ff35f319/volumes"
Jul 28 11:25:40.339311 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:40.339281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerStarted","Data":"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"}
Jul 28 11:25:41.345769 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:41.345732    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerStarted","Data":"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"}
Jul 28 11:25:41.346267 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:41.345850    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:41.378095 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:41.378045    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podStartSLOduration=3.110494962 podStartE2EDuration="2m1.378029859s" totalImagesPullingTime="1m54.459408649s" totalInitContainerRuntime="3.808126248s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:25:41.376167099 +0000 UTC m=+4699.296871150" watchObservedRunningTime="2026-07-28 11:25:41.378029859 +0000 UTC m=+4699.298733911"
Jul 28 11:25:42.350095 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:42.350054    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:42.351204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:42.351178    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Jul 28 11:25:43.353522 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:43.353484    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.71:8080: connect: connection refused"
Jul 28 11:25:48.358532 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:48.358504    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:48.359241 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:48.359223    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:51.667131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:51.667097    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:25:51.667513 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:51.667332    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container" containerID="cri-o://7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e" gracePeriod=30
Jul 28 11:25:51.667513 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:51.667363    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kube-rbac-proxy" containerID="cri-o://84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39" gracePeriod=30
Jul 28 11:25:52.392614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:52.392580    2569 generic.go:356] "Generic (PLEG): container finished" podID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerID="84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39" exitCode=2
Jul 28 11:25:52.392811 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:52.392652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerDied","Data":"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"}
Jul 28 11:25:53.354549 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:53.354508    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.71:8643/healthz\": dial tcp 10.134.0.71:8643: connect: connection refused"
Jul 28 11:25:54.137140 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.137114    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:54.149763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.149713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location\") pod \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") "
Jul 28 11:25:54.149763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.149746    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9hd8\" (UniqueName: \"kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8\") pod \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") "
Jul 28 11:25:54.149909 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.149791    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") pod \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") "
Jul 28 11:25:54.149909 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.149819    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config\") pod \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") "
Jul 28 11:25:54.149909 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.149840    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert\") pod \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\" (UID: \"883a7475-e4ff-48f0-9380-e4ec60f5f24e\") "
Jul 28 11:25:54.150200 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.150176    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "883a7475-e4ff-48f0-9380-e4ec60f5f24e" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:25:54.150276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.150208    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "883a7475-e4ff-48f0-9380-e4ec60f5f24e" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:25:54.150276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.150225    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "883a7475-e4ff-48f0-9380-e4ec60f5f24e" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:25:54.151741 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.151722    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8" (OuterVolumeSpecName: "kube-api-access-h9hd8") pod "883a7475-e4ff-48f0-9380-e4ec60f5f24e" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e"). InnerVolumeSpecName "kube-api-access-h9hd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:25:54.151836 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.151825    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "883a7475-e4ff-48f0-9380-e4ec60f5f24e" (UID: "883a7475-e4ff-48f0-9380-e4ec60f5f24e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:25:54.250340 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.250305    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:25:54.250340 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.250333    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/883a7475-e4ff-48f0-9380-e4ec60f5f24e-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:25:54.250340 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.250342    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:25:54.250578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.250351    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hd8\" (UniqueName: \"kubernetes.io/projected/883a7475-e4ff-48f0-9380-e4ec60f5f24e-kube-api-access-h9hd8\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:25:54.250578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.250360    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/883a7475-e4ff-48f0-9380-e4ec60f5f24e-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:25:54.402385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.402298    2569 generic.go:356] "Generic (PLEG): container finished" podID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerID="7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e" exitCode=0
Jul 28 11:25:54.402385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.402339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerDied","Data":"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"}
Jul 28 11:25:54.402884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.402418    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b" event={"ID":"883a7475-e4ff-48f0-9380-e4ec60f5f24e","Type":"ContainerDied","Data":"52734d3ab145d217b9c8d40bcf5f455542dcc00f91b238e9465bd9578a2a9dcd"}
Jul 28 11:25:54.402884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.402362    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"
Jul 28 11:25:54.402884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.402454    2569 scope.go:117] "RemoveContainer" containerID="84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"
Jul 28 11:25:54.412117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.412099    2569 scope.go:117] "RemoveContainer" containerID="7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"
Jul 28 11:25:54.420094 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.420073    2569 scope.go:117] "RemoveContainer" containerID="62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"
Jul 28 11:25:54.425878 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.425850    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:25:54.428498 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.428475    2569 scope.go:117] "RemoveContainer" containerID="84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"
Jul 28 11:25:54.428931 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:25:54.428906    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39\": container with ID starting with 84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39 not found: ID does not exist" containerID="84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"
Jul 28 11:25:54.429022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.428943    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39"} err="failed to get container status \"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39\": rpc error: code = NotFound desc = could not find container \"84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39\": container with ID starting with 84da7afb67c97d5133beb7b6f1c5bf4a01c83b7045b8e4df6f4cbb6f78373a39 not found: ID does not exist"
Jul 28 11:25:54.429022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.428964    2569 scope.go:117] "RemoveContainer" containerID="7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"
Jul 28 11:25:54.429259 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:25:54.429244    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e\": container with ID starting with 7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e not found: ID does not exist" containerID="7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"
Jul 28 11:25:54.429302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.429263    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e"} err="failed to get container status \"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e\": rpc error: code = NotFound desc = could not find container \"7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e\": container with ID starting with 7eb81c4d3dc64075923c994ddea433c991d980deae47fcbe2c90ffeb93aab02e not found: ID does not exist"
Jul 28 11:25:54.429302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.429281    2569 scope.go:117] "RemoveContainer" containerID="62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"
Jul 28 11:25:54.429527 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:25:54.429505    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2\": container with ID starting with 62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2 not found: ID does not exist" containerID="62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"
Jul 28 11:25:54.429579 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.429535    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2"} err="failed to get container status \"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2\": rpc error: code = NotFound desc = could not find container \"62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2\": container with ID starting with 62277c022c06014cc72a2bcd1e6f950ef4d9068611765ee7c184dede9258c0f2 not found: ID does not exist"
Jul 28 11:25:54.429779 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.429754    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-rjs9b"]
Jul 28 11:25:54.565359 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:54.565326    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" path="/var/lib/kubelet/pods/883a7475-e4ff-48f0-9380-e4ec60f5f24e/volumes"
Jul 28 11:25:56.805810 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.805774    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806061    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806072    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806082    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806088    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806105    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806111    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806117    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="storage-initializer"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806123    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="storage-initializer"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806129    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="storage-initializer"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806136    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="storage-initializer"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806150    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806195    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kserve-container"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806204    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806211    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7df6eb1-0ff4-4b50-946b-1588ff35f319" containerName="kube-rbac-proxy"
Jul 28 11:25:56.819310 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.806217    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="883a7475-e4ff-48f0-9380-e4ec60f5f24e" containerName="kserve-container"
Jul 28 11:25:56.827297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.827271    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:25:56.827442 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.827420    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.830785 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.830760    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:25:56.830918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.830770    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 11:25:56.830918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.830815    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:25:56.830918 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.830871    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:25:56.831858 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.831839    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:25:56.831858 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.831849    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 11:25:56.832014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.831839    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:25:56.869117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.869090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.869242 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.869136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.869242 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.869163    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.869242 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.869185    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.869242 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.869208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gzk\" (UniqueName: \"kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.970373 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.970373 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970377    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.970658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.970658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gzk\" (UniqueName: \"kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.970658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.971017 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.970995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.971295 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.971271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.971342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.971271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.972895 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.972872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:56.978586 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:56.978564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gzk\" (UniqueName: \"kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk\") pod \"isvc-xgboost-predictor-7566d65bf7-48bb6\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:57.138035 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:57.137940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:25:57.297568 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:57.297540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:25:57.300243 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:25:57.300219    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e13b163_fb82_4abe_9f37_4b8ead28522e.slice/crio-39f9ebccc4118ea332cd2cd149a8aab95af47b2118f024b58bdb579295310880 WatchSource:0}: Error finding container 39f9ebccc4118ea332cd2cd149a8aab95af47b2118f024b58bdb579295310880: Status 404 returned error can't find the container with id 39f9ebccc4118ea332cd2cd149a8aab95af47b2118f024b58bdb579295310880
Jul 28 11:25:57.416842 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:57.416806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerStarted","Data":"39f9ebccc4118ea332cd2cd149a8aab95af47b2118f024b58bdb579295310880"}
Jul 28 11:25:58.422011 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:25:58.421979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerStarted","Data":"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"}
Jul 28 11:26:01.435034 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:01.435001    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerID="0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527" exitCode=0
Jul 28 11:26:01.435459 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:01.435078    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerDied","Data":"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"}
Jul 28 11:26:19.520004 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:19.519916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerStarted","Data":"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"}
Jul 28 11:26:19.520004 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:19.519979    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerStarted","Data":"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"}
Jul 28 11:26:19.520480 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:19.520161    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:26:19.541187 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:19.541144    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podStartSLOduration=1.9055379430000001 podStartE2EDuration="23.541129784s" totalImagesPullingTime="17.800417123s" totalInitContainerRuntime="3.835174718s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:26:19.538390211 +0000 UTC m=+4737.459094265" watchObservedRunningTime="2026-07-28 11:26:19.541129784 +0000 UTC m=+4737.461833835"
Jul 28 11:26:20.523990 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:20.523956    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:26:20.524971 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:20.524947    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:26:21.528117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:21.528078    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:26:26.533034 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:26.533004    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:26:26.533507 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:26.533453    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:26:36.533671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:36.533628    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:26:46.533838 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:46.533746    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:26:56.534420 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:26:56.534380    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:27:06.534320 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:06.534273    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:27:16.534409 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:16.534367    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Jul 28 11:27:22.809837 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:22.809809    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:27:22.824168 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:22.824141    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:27:26.534603 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:26.534574    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:27:36.861866 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:36.861764    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:27:36.862302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:36.862019    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container" containerID="cri-o://dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731" gracePeriod=30
Jul 28 11:27:36.862302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:36.862057    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kube-rbac-proxy" containerID="cri-o://f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801" gracePeriod=30
Jul 28 11:27:37.821638 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:37.821604    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerID="f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801" exitCode=2
Jul 28 11:27:37.821834 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:37.821674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerDied","Data":"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"}
Jul 28 11:27:40.605238 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.605214    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:27:40.732155 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732119    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location\") pod \"3e13b163-fb82-4abe-9f37-4b8ead28522e\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") "
Jul 28 11:27:40.732155 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732156    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5gzk\" (UniqueName: \"kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk\") pod \"3e13b163-fb82-4abe-9f37-4b8ead28522e\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") "
Jul 28 11:27:40.732426 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732177    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls\") pod \"3e13b163-fb82-4abe-9f37-4b8ead28522e\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") "
Jul 28 11:27:40.732426 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732205    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config\") pod \"3e13b163-fb82-4abe-9f37-4b8ead28522e\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") "
Jul 28 11:27:40.732426 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732245    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert\") pod \"3e13b163-fb82-4abe-9f37-4b8ead28522e\" (UID: \"3e13b163-fb82-4abe-9f37-4b8ead28522e\") "
Jul 28 11:27:40.732595 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732509    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e13b163-fb82-4abe-9f37-4b8ead28522e" (UID: "3e13b163-fb82-4abe-9f37-4b8ead28522e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:27:40.732665 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732637    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e13b163-fb82-4abe-9f37-4b8ead28522e" (UID: "3e13b163-fb82-4abe-9f37-4b8ead28522e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:27:40.732749 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.732668    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e13b163-fb82-4abe-9f37-4b8ead28522e" (UID: "3e13b163-fb82-4abe-9f37-4b8ead28522e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:27:40.734302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.734274    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e13b163-fb82-4abe-9f37-4b8ead28522e" (UID: "3e13b163-fb82-4abe-9f37-4b8ead28522e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:27:40.734302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.734279    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk" (OuterVolumeSpecName: "kube-api-access-h5gzk") pod "3e13b163-fb82-4abe-9f37-4b8ead28522e" (UID: "3e13b163-fb82-4abe-9f37-4b8ead28522e"). InnerVolumeSpecName "kube-api-access-h5gzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:27:40.833614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.833575    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e13b163-fb82-4abe-9f37-4b8ead28522e-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:27:40.833614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.833604    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gzk\" (UniqueName: \"kubernetes.io/projected/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-api-access-h5gzk\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:27:40.833614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.833615    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e13b163-fb82-4abe-9f37-4b8ead28522e-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:27:40.833931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.833628    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:27:40.833931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.833641    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e13b163-fb82-4abe-9f37-4b8ead28522e-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:27:40.835255 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.835220    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerID="dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731" exitCode=0
Jul 28 11:27:40.835392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.835255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerDied","Data":"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"}
Jul 28 11:27:40.835392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.835288    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"
Jul 28 11:27:40.835392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.835337    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6" event={"ID":"3e13b163-fb82-4abe-9f37-4b8ead28522e","Type":"ContainerDied","Data":"39f9ebccc4118ea332cd2cd149a8aab95af47b2118f024b58bdb579295310880"}
Jul 28 11:27:40.835392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.835375    2569 scope.go:117] "RemoveContainer" containerID="f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"
Jul 28 11:27:40.843825 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.843807    2569 scope.go:117] "RemoveContainer" containerID="dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"
Jul 28 11:27:40.851620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.851595    2569 scope.go:117] "RemoveContainer" containerID="0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"
Jul 28 11:27:40.860961 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.860854    2569 scope.go:117] "RemoveContainer" containerID="f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"
Jul 28 11:27:40.861470 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:27:40.861355    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801\": container with ID starting with f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801 not found: ID does not exist" containerID="f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"
Jul 28 11:27:40.861470 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.861420    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801"} err="failed to get container status \"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801\": rpc error: code = NotFound desc = could not find container \"f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801\": container with ID starting with f996c61e675ec1ea70caccf9bad70577edea774e7747b97c1501a9d6bb3d4801 not found: ID does not exist"
Jul 28 11:27:40.861470 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.861445    2569 scope.go:117] "RemoveContainer" containerID="dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"
Jul 28 11:27:40.861736 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.861486    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:27:40.861874 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:27:40.861848    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731\": container with ID starting with dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731 not found: ID does not exist" containerID="dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"
Jul 28 11:27:40.861925 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.861883    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731"} err="failed to get container status \"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731\": rpc error: code = NotFound desc = could not find container \"dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731\": container with ID starting with dc082344f7cf22811b6134b5faf14aed5659a80bfa11b972033eee8878f63731 not found: ID does not exist"
Jul 28 11:27:40.861925 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.861906    2569 scope.go:117] "RemoveContainer" containerID="0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"
Jul 28 11:27:40.862221 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:27:40.862194    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527\": container with ID starting with 0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527 not found: ID does not exist" containerID="0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"
Jul 28 11:27:40.862284 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.862224    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527"} err="failed to get container status \"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527\": rpc error: code = NotFound desc = could not find container \"0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527\": container with ID starting with 0c9a43a7d4867d7aabd3ffa1806a0a9283dcc697612fabbf6a1d2a09b640d527 not found: ID does not exist"
Jul 28 11:27:40.863116 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:40.863096    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-48bb6"]
Jul 28 11:27:42.052431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052397    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052677    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kube-rbac-proxy"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052702    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kube-rbac-proxy"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052719    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="storage-initializer"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052726    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="storage-initializer"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052740    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052746    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052796    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kserve-container"
Jul 28 11:27:42.052846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.052805    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" containerName="kube-rbac-proxy"
Jul 28 11:27:42.055657 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.055639    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.062424 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.062404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:27:42.062607 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.062588    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:27:42.062724 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.062593    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 11:27:42.063194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.063177    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 11:27:42.063531 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.063515    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:27:42.064322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.064302    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:27:42.064414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.064330    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:27:42.090016 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.089981    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:27:42.145358 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.145326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6gw\" (UniqueName: \"kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.145556 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.145374    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.145556 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.145502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.145556 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.145545    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.145679 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.145566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6gw\" (UniqueName: \"kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246318    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246561 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246561 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246413    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246561 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.246863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.246842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.247152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.247130    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.247219 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.247180    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.248907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.248891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.255165 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.255134    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6gw\" (UniqueName: \"kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.365703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.365599    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:42.494923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.494900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:27:42.497192 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:27:42.497165    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edd9ded_542e_4317_abd9_402a56f91d30.slice/crio-a5cb97cda523d4db2381c96ef0a8d10410ce2b325a021f0da24b6bb0d2579e93 WatchSource:0}: Error finding container a5cb97cda523d4db2381c96ef0a8d10410ce2b325a021f0da24b6bb0d2579e93: Status 404 returned error can't find the container with id a5cb97cda523d4db2381c96ef0a8d10410ce2b325a021f0da24b6bb0d2579e93
Jul 28 11:27:42.566507 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.566482    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e13b163-fb82-4abe-9f37-4b8ead28522e" path="/var/lib/kubelet/pods/3e13b163-fb82-4abe-9f37-4b8ead28522e/volumes"
Jul 28 11:27:42.846058 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.846015    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerStarted","Data":"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"}
Jul 28 11:27:42.846240 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:42.846076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerStarted","Data":"a5cb97cda523d4db2381c96ef0a8d10410ce2b325a021f0da24b6bb0d2579e93"}
Jul 28 11:27:46.865623 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:46.865589    2569 generic.go:356] "Generic (PLEG): container finished" podID="3edd9ded-542e-4317-abd9-402a56f91d30" containerID="c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4" exitCode=0
Jul 28 11:27:46.866022 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:46.865661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerDied","Data":"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"}
Jul 28 11:27:47.871716 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:47.871666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerStarted","Data":"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"}
Jul 28 11:27:47.872183 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:47.871745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerStarted","Data":"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"}
Jul 28 11:27:47.872183 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:47.871950    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:47.895064 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:47.895001    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" podStartSLOduration=2.154846254 podStartE2EDuration="5.894979431s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.740133177s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:27:47.892867732 +0000 UTC m=+4825.813571795" watchObservedRunningTime="2026-07-28 11:27:47.894979431 +0000 UTC m=+4825.815683480"
Jul 28 11:27:48.876165 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:48.876134    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:27:54.886005 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:27:54.885977    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:28:24.890306 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:24.890276    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:28:32.088728 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:32.088695    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:28:32.089138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:32.088922    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kserve-container" containerID="cri-o://7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f" gracePeriod=30
Jul 28 11:28:32.089138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:32.088971    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kube-rbac-proxy" containerID="cri-o://dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3" gracePeriod=30
Jul 28 11:28:33.054848 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:33.054808    2569 generic.go:356] "Generic (PLEG): container finished" podID="3edd9ded-542e-4317-abd9-402a56f91d30" containerID="dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3" exitCode=2
Jul 28 11:28:33.055027 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:33.054879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerDied","Data":"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"}
Jul 28 11:28:34.881025 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:34.880987    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.73:8643/healthz\": dial tcp 10.134.0.73:8643: connect: connection refused"
Jul 28 11:28:34.886545 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:34.886515    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.73:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.73:8080: connect: connection refused"
Jul 28 11:28:37.280658 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.280620    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:28:37.284171 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.284152    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.286734 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.286713    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 11:28:37.286734 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.286729    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 11:28:37.294578 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.294556    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:28:37.400222 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.400183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.400222 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.400228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.400447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.400305    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.400447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.400370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47s8t\" (UniqueName: \"kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.400447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.400416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501189 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501189 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501192    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501240    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47s8t\" (UniqueName: \"kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501432 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.501652 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.502007 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.501990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.502050 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.502008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.503614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.503592    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.510377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.510353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47s8t\" (UniqueName: \"kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-s98b7\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.594562 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.594475    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:37.726409 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:37.726385    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:28:37.728463 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:28:37.728439    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24968a88_0180_4677_aead_20f68350857f.slice/crio-367a6ed998253c060f434035c383032de23652d3d8b803809ffb835ce814cb96 WatchSource:0}: Error finding container 367a6ed998253c060f434035c383032de23652d3d8b803809ffb835ce814cb96: Status 404 returned error can't find the container with id 367a6ed998253c060f434035c383032de23652d3d8b803809ffb835ce814cb96
Jul 28 11:28:38.076759 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.076717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerStarted","Data":"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"}
Jul 28 11:28:38.076954 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.076790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerStarted","Data":"367a6ed998253c060f434035c383032de23652d3d8b803809ffb835ce814cb96"}
Jul 28 11:28:38.728191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.728164    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:28:38.812332 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812238    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location\") pod \"3edd9ded-542e-4317-abd9-402a56f91d30\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") "
Jul 28 11:28:38.812332 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812286    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls\") pod \"3edd9ded-542e-4317-abd9-402a56f91d30\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") "
Jul 28 11:28:38.812332 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config\") pod \"3edd9ded-542e-4317-abd9-402a56f91d30\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") "
Jul 28 11:28:38.812594 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812342    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb6gw\" (UniqueName: \"kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw\") pod \"3edd9ded-542e-4317-abd9-402a56f91d30\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") "
Jul 28 11:28:38.812594 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812373    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert\") pod \"3edd9ded-542e-4317-abd9-402a56f91d30\" (UID: \"3edd9ded-542e-4317-abd9-402a56f91d30\") "
Jul 28 11:28:38.812731 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812589    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3edd9ded-542e-4317-abd9-402a56f91d30" (UID: "3edd9ded-542e-4317-abd9-402a56f91d30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:28:38.812807 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812787    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3edd9ded-542e-4317-abd9-402a56f91d30-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:28:38.812935 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3edd9ded-542e-4317-abd9-402a56f91d30" (UID: "3edd9ded-542e-4317-abd9-402a56f91d30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:28:38.813017 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.812910    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3edd9ded-542e-4317-abd9-402a56f91d30" (UID: "3edd9ded-542e-4317-abd9-402a56f91d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:28:38.814552 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.814524    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3edd9ded-542e-4317-abd9-402a56f91d30" (UID: "3edd9ded-542e-4317-abd9-402a56f91d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:28:38.814665 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.814561    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw" (OuterVolumeSpecName: "kube-api-access-mb6gw") pod "3edd9ded-542e-4317-abd9-402a56f91d30" (UID: "3edd9ded-542e-4317-abd9-402a56f91d30"). InnerVolumeSpecName "kube-api-access-mb6gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:28:38.914153 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.914099    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3edd9ded-542e-4317-abd9-402a56f91d30-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:28:38.914153 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.914146    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:28:38.914153 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.914157    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb6gw\" (UniqueName: \"kubernetes.io/projected/3edd9ded-542e-4317-abd9-402a56f91d30-kube-api-access-mb6gw\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:28:38.914153 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:38.914166    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3edd9ded-542e-4317-abd9-402a56f91d30-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:28:39.082748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.082642    2569 generic.go:356] "Generic (PLEG): container finished" podID="3edd9ded-542e-4317-abd9-402a56f91d30" containerID="7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f" exitCode=0
Jul 28 11:28:39.082748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.082724    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"
Jul 28 11:28:39.082748 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.082730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerDied","Data":"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"}
Jul 28 11:28:39.082948 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.082829    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g" event={"ID":"3edd9ded-542e-4317-abd9-402a56f91d30","Type":"ContainerDied","Data":"a5cb97cda523d4db2381c96ef0a8d10410ce2b325a021f0da24b6bb0d2579e93"}
Jul 28 11:28:39.082948 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.082857    2569 scope.go:117] "RemoveContainer" containerID="dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"
Jul 28 11:28:39.091267 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.091251    2569 scope.go:117] "RemoveContainer" containerID="7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"
Jul 28 11:28:39.098845 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.098820    2569 scope.go:117] "RemoveContainer" containerID="c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"
Jul 28 11:28:39.105525 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.105500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:28:39.106286 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.106274    2569 scope.go:117] "RemoveContainer" containerID="dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"
Jul 28 11:28:39.106529 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:28:39.106508    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3\": container with ID starting with dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3 not found: ID does not exist" containerID="dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"
Jul 28 11:28:39.106611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.106535    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3"} err="failed to get container status \"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3\": rpc error: code = NotFound desc = could not find container \"dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3\": container with ID starting with dd91c0fd92e6ca8ec81d6281488efb043e54539015d9fdd1a6e4e376f5e185d3 not found: ID does not exist"
Jul 28 11:28:39.106611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.106553    2569 scope.go:117] "RemoveContainer" containerID="7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"
Jul 28 11:28:39.106846 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:28:39.106829    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f\": container with ID starting with 7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f not found: ID does not exist" containerID="7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"
Jul 28 11:28:39.106912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.106850    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f"} err="failed to get container status \"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f\": rpc error: code = NotFound desc = could not find container \"7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f\": container with ID starting with 7064b6b5ef8478e4bedd8e93b304d191e58d8113642540cd653ced4cbdeb536f not found: ID does not exist"
Jul 28 11:28:39.106912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.106866    2569 scope.go:117] "RemoveContainer" containerID="c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"
Jul 28 11:28:39.107115 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:28:39.107080    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4\": container with ID starting with c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4 not found: ID does not exist" containerID="c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"
Jul 28 11:28:39.107188 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.107114    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4"} err="failed to get container status \"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4\": rpc error: code = NotFound desc = could not find container \"c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4\": container with ID starting with c8b808761389cbc17617862c7c8e0e0095189a52361810c602df38b841cc7ff4 not found: ID does not exist"
Jul 28 11:28:39.109342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:39.109316    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tb78g"]
Jul 28 11:28:40.566179 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:40.566148    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" path="/var/lib/kubelet/pods/3edd9ded-542e-4317-abd9-402a56f91d30/volumes"
Jul 28 11:28:42.096912 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:42.096879    2569 generic.go:356] "Generic (PLEG): container finished" podID="24968a88-0180-4677-aead-20f68350857f" containerID="d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6" exitCode=0
Jul 28 11:28:42.097300 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:42.096947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerDied","Data":"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"}
Jul 28 11:28:43.103442 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:43.103411    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerStarted","Data":"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"}
Jul 28 11:28:43.103863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:43.103470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerStarted","Data":"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"}
Jul 28 11:28:43.103863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:43.103757    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:43.103863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:43.103824    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:28:43.128249 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:43.128204    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" podStartSLOduration=2.57028316 podStartE2EDuration="6.12819138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.55790822s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:28:43.126716166 +0000 UTC m=+4881.047420219" watchObservedRunningTime="2026-07-28 11:28:43.12819138 +0000 UTC m=+4881.048895491"
Jul 28 11:28:49.112421 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:28:49.112391    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:29:19.116967 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:19.116929    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:29:27.332526 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:27.332487    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:29:27.333014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:27.332799    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kserve-container" containerID="cri-o://da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6" gracePeriod=30
Jul 28 11:29:27.333014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:27.332835    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kube-rbac-proxy" containerID="cri-o://8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785" gracePeriod=30
Jul 28 11:29:28.295582 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:28.295547    2569 generic.go:356] "Generic (PLEG): container finished" podID="24968a88-0180-4677-aead-20f68350857f" containerID="8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785" exitCode=2
Jul 28 11:29:28.295785 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:28.295586    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerDied","Data":"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"}
Jul 28 11:29:29.108696 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:29.108651    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.74:8643/healthz\": dial tcp 10.134.0.74:8643: connect: connection refused"
Jul 28 11:29:32.519362 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519324    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519698    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kserve-container"
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519712    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kserve-container"
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519735    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kube-rbac-proxy"
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519741    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kube-rbac-proxy"
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519749    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="storage-initializer"
Jul 28 11:29:32.519776 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519755    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="storage-initializer"
Jul 28 11:29:32.520010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519803    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kserve-container"
Jul 28 11:29:32.520010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.519811    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edd9ded-542e-4317-abd9-402a56f91d30" containerName="kube-rbac-proxy"
Jul 28 11:29:32.522910 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.522895    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.525648 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.525621    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 11:29:32.525648 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.525631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 11:29:32.533972 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.533949    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:29:32.645846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.645814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.646038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.645881    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.646038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.645960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wgk\" (UniqueName: \"kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.646038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.645992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.646038 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.646032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747327 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747289    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wgk\" (UniqueName: \"kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.747879 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.747675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.748165 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.748143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.748239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.748146    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.749598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.749581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.756357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.756328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wgk\" (UniqueName: \"kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-x6hgm\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.832875 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.832787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:32.964819 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:32.964794    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:29:32.967315 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:29:32.967286    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2dd96e_e65b_4d2b_ad6c_49b3367c7f15.slice/crio-3e6818ce99f94655211cce4abac4342591756680dd5f407c325dccb8b410debf WatchSource:0}: Error finding container 3e6818ce99f94655211cce4abac4342591756680dd5f407c325dccb8b410debf: Status 404 returned error can't find the container with id 3e6818ce99f94655211cce4abac4342591756680dd5f407c325dccb8b410debf
Jul 28 11:29:33.318193 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.318149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerStarted","Data":"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"}
Jul 28 11:29:33.318380 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.318210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerStarted","Data":"3e6818ce99f94655211cce4abac4342591756680dd5f407c325dccb8b410debf"}
Jul 28 11:29:33.875964 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.875942    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:29:33.956815 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.956726    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert\") pod \"24968a88-0180-4677-aead-20f68350857f\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") "
Jul 28 11:29:33.956815 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.956779    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location\") pod \"24968a88-0180-4677-aead-20f68350857f\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") "
Jul 28 11:29:33.956815 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.956814    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config\") pod \"24968a88-0180-4677-aead-20f68350857f\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") "
Jul 28 11:29:33.957108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.956845    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls\") pod \"24968a88-0180-4677-aead-20f68350857f\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") "
Jul 28 11:29:33.957108 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.956867    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47s8t\" (UniqueName: \"kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t\") pod \"24968a88-0180-4677-aead-20f68350857f\" (UID: \"24968a88-0180-4677-aead-20f68350857f\") "
Jul 28 11:29:33.957217 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.957122    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24968a88-0180-4677-aead-20f68350857f" (UID: "24968a88-0180-4677-aead-20f68350857f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:29:33.957217 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.957152    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24968a88-0180-4677-aead-20f68350857f" (UID: "24968a88-0180-4677-aead-20f68350857f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:29:33.957217 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.957198    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24968a88-0180-4677-aead-20f68350857f" (UID: "24968a88-0180-4677-aead-20f68350857f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:29:33.958982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.958951    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24968a88-0180-4677-aead-20f68350857f" (UID: "24968a88-0180-4677-aead-20f68350857f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:29:33.959060 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:33.958976    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t" (OuterVolumeSpecName: "kube-api-access-47s8t") pod "24968a88-0180-4677-aead-20f68350857f" (UID: "24968a88-0180-4677-aead-20f68350857f"). InnerVolumeSpecName "kube-api-access-47s8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:29:34.057796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.057766    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:29:34.057796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.057793    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24968a88-0180-4677-aead-20f68350857f-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:29:34.057796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.057803    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24968a88-0180-4677-aead-20f68350857f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:29:34.058033 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.057814    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24968a88-0180-4677-aead-20f68350857f-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:29:34.058033 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.057824    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47s8t\" (UniqueName: \"kubernetes.io/projected/24968a88-0180-4677-aead-20f68350857f-kube-api-access-47s8t\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:29:34.324587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.324498    2569 generic.go:356] "Generic (PLEG): container finished" podID="24968a88-0180-4677-aead-20f68350857f" containerID="da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6" exitCode=0
Jul 28 11:29:34.324587 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.324562    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"
Jul 28 11:29:34.324803 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.324586    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerDied","Data":"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"}
Jul 28 11:29:34.324803 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.324650    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7" event={"ID":"24968a88-0180-4677-aead-20f68350857f","Type":"ContainerDied","Data":"367a6ed998253c060f434035c383032de23652d3d8b803809ffb835ce814cb96"}
Jul 28 11:29:34.324803 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.324680    2569 scope.go:117] "RemoveContainer" containerID="8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"
Jul 28 11:29:34.333596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.333578    2569 scope.go:117] "RemoveContainer" containerID="da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"
Jul 28 11:29:34.340778 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.340758    2569 scope.go:117] "RemoveContainer" containerID="d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"
Jul 28 11:29:34.347282 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.347259    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:29:34.348042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.348024    2569 scope.go:117] "RemoveContainer" containerID="8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"
Jul 28 11:29:34.348302 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:29:34.348281    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785\": container with ID starting with 8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785 not found: ID does not exist" containerID="8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"
Jul 28 11:29:34.348359 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.348312    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785"} err="failed to get container status \"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785\": rpc error: code = NotFound desc = could not find container \"8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785\": container with ID starting with 8c6153b437bc735836c6eae5fe38a1aea4186aaf4c582405cacb35ece2a1d785 not found: ID does not exist"
Jul 28 11:29:34.348359 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.348331    2569 scope.go:117] "RemoveContainer" containerID="da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"
Jul 28 11:29:34.348609 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:29:34.348587    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6\": container with ID starting with da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6 not found: ID does not exist" containerID="da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"
Jul 28 11:29:34.348720 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.348615    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6"} err="failed to get container status \"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6\": rpc error: code = NotFound desc = could not find container \"da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6\": container with ID starting with da889103bf16741c96fb6350884fb050c66410afa73067aa3d243b5a6d5beaf6 not found: ID does not exist"
Jul 28 11:29:34.348720 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.348636    2569 scope.go:117] "RemoveContainer" containerID="d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"
Jul 28 11:29:34.349115 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:29:34.349095    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6\": container with ID starting with d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6 not found: ID does not exist" containerID="d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"
Jul 28 11:29:34.349226 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.349124    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6"} err="failed to get container status \"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6\": rpc error: code = NotFound desc = could not find container \"d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6\": container with ID starting with d112d057f009389a1f8bc560f39edbbd0b7a30d9a9a821b64b98622ec7ecbfc6 not found: ID does not exist"
Jul 28 11:29:34.350817 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.350801    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-s98b7"]
Jul 28 11:29:34.566023 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:34.565993    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24968a88-0180-4677-aead-20f68350857f" path="/var/lib/kubelet/pods/24968a88-0180-4677-aead-20f68350857f/volumes"
Jul 28 11:29:37.338317 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:37.338283    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerID="904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84" exitCode=0
Jul 28 11:29:37.338722 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:37.338355    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerDied","Data":"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"}
Jul 28 11:29:38.344334 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.344297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerStarted","Data":"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"}
Jul 28 11:29:38.344750 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.344358    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerStarted","Data":"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"}
Jul 28 11:29:38.344750 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.344624    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:38.344849 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.344795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:38.346129 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.346103    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:29:38.364774 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:38.364729    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podStartSLOduration=2.6850822 podStartE2EDuration="6.364715445s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.679633245s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:29:38.362951922 +0000 UTC m=+4936.283655974" watchObservedRunningTime="2026-07-28 11:29:38.364715445 +0000 UTC m=+4936.285419497"
Jul 28 11:29:39.348230 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:39.348195    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:29:44.353547 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:44.353464    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:29:44.354004 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:44.353979    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:29:54.354473 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:29:54.354436    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:04.354863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:04.354821    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:14.354822 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:14.354783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:24.354881 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:24.354840    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:34.354939 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:34.354895    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:44.354843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:44.354813    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:30:52.593414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:52.593380    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:30:52.593879 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:52.593639    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" containerID="cri-o://794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d" gracePeriod=30
Jul 28 11:30:52.593879 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:52.593659    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kube-rbac-proxy" containerID="cri-o://429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb" gracePeriod=30
Jul 28 11:30:53.641702 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:53.641663    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerID="429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb" exitCode=2
Jul 28 11:30:53.642079 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:53.641723    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerDied","Data":"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"}
Jul 28 11:30:54.349040 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:54.349005    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.75:8643/healthz\": dial tcp 10.134.0.75:8643: connect: connection refused"
Jul 28 11:30:54.354274 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:54.354250    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.75:8080: connect: connection refused"
Jul 28 11:30:56.236832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.236802    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:30:56.332200 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2wgk\" (UniqueName: \"kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk\") pod \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") "
Jul 28 11:30:56.332394 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332211    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location\") pod \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") "
Jul 28 11:30:56.332394 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332233    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert\") pod \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") "
Jul 28 11:30:56.332394 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332253    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls\") pod \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") "
Jul 28 11:30:56.332597 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332415    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config\") pod \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\" (UID: \"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15\") "
Jul 28 11:30:56.332597 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332543    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" (UID: "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:30:56.332736 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332630    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" (UID: "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:30:56.332794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332742    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:30:56.332794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332761    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:30:56.332794 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.332781    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" (UID: "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:30:56.334278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.334257    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" (UID: "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:30:56.334380 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.334364    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk" (OuterVolumeSpecName: "kube-api-access-v2wgk") pod "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" (UID: "3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15"). InnerVolumeSpecName "kube-api-access-v2wgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:30:56.433223 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.433191    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2wgk\" (UniqueName: \"kubernetes.io/projected/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-api-access-v2wgk\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:30:56.433223 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.433219    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:30:56.433223 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.433228    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:30:56.656909 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.656877    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerID="794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d" exitCode=0
Jul 28 11:30:56.657077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.656932    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"
Jul 28 11:30:56.657077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.656956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerDied","Data":"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"}
Jul 28 11:30:56.657077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.657013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm" event={"ID":"3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15","Type":"ContainerDied","Data":"3e6818ce99f94655211cce4abac4342591756680dd5f407c325dccb8b410debf"}
Jul 28 11:30:56.657077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.657039    2569 scope.go:117] "RemoveContainer" containerID="429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"
Jul 28 11:30:56.665122 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.665105    2569 scope.go:117] "RemoveContainer" containerID="794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"
Jul 28 11:30:56.672088 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.672071    2569 scope.go:117] "RemoveContainer" containerID="904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"
Jul 28 11:30:56.674305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.674285    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:30:56.679712 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.679675    2569 scope.go:117] "RemoveContainer" containerID="429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"
Jul 28 11:30:56.679981 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:30:56.679961    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb\": container with ID starting with 429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb not found: ID does not exist" containerID="429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"
Jul 28 11:30:56.680048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.679990    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb"} err="failed to get container status \"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb\": rpc error: code = NotFound desc = could not find container \"429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb\": container with ID starting with 429966e26691374d0375665c4f6f48950d8deed538eb7ac4c50051e9b4a04bdb not found: ID does not exist"
Jul 28 11:30:56.680048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.680009    2569 scope.go:117] "RemoveContainer" containerID="794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"
Jul 28 11:30:56.680275 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:30:56.680258    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d\": container with ID starting with 794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d not found: ID does not exist" containerID="794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"
Jul 28 11:30:56.680330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.680280    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d"} err="failed to get container status \"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d\": rpc error: code = NotFound desc = could not find container \"794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d\": container with ID starting with 794d497611c211aa1510528a019d8804a1739d4b6785bfc7cbbf02163fa5cb8d not found: ID does not exist"
Jul 28 11:30:56.680330 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.680296    2569 scope.go:117] "RemoveContainer" containerID="904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"
Jul 28 11:30:56.680523 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:30:56.680504    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84\": container with ID starting with 904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84 not found: ID does not exist" containerID="904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"
Jul 28 11:30:56.680568 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.680526    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-x6hgm"]
Jul 28 11:30:56.680568 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:56.680528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84"} err="failed to get container status \"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84\": rpc error: code = NotFound desc = could not find container \"904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84\": container with ID starting with 904f56d54d766f798b05d92ddd6dd3ecd795b2986026bdf37b184c52eedf7f84 not found: ID does not exist"
Jul 28 11:30:57.761841 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.761808    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762108    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762120    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762129    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kube-rbac-proxy"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762136    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kube-rbac-proxy"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762143    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kube-rbac-proxy"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762149    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kube-rbac-proxy"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762157    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="storage-initializer"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762162    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="storage-initializer"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762175    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="storage-initializer"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762179    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="storage-initializer"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762187    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762192    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762237    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762246    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kserve-container"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762252    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" containerName="kube-rbac-proxy"
Jul 28 11:30:57.764203 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.762258    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24968a88-0180-4677-aead-20f68350857f" containerName="kube-rbac-proxy"
Jul 28 11:30:57.765076 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.765062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.767583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 11:30:57.767783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767619    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:30:57.767783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767638    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:30:57.767783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767618    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 11:30:57.767783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:30:57.767783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.767665    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:30:57.768453 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.768439    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 11:30:57.776425 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.776405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:30:57.842301 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.842250    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.842464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.842307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5x9\" (UniqueName: \"kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.842464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.842367    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.842464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.842402    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.842464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.842442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.943634    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.943673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.943722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.943767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943926 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.943793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5x9\" (UniqueName: \"kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.943926 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:30:57.943877    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 11:30:57.944141 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:30:57.943946    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls podName:7955e3e8-d873-4c53-ac2b-96d59a1709f6 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:58.443925065 +0000 UTC m=+5016.364629099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 11:30:57.944141 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.944040    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.944395 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.944372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.944500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.944482    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:57.952491 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:57.952471    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5x9\" (UniqueName: \"kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:58.447580 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:58.447539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:58.450087 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:58.450051    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:58.565336 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:58.565306    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15" path="/var/lib/kubelet/pods/3b2dd96e-e65b-4d2b-ad6c-49b3367c7f15/volumes"
Jul 28 11:30:58.675533 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:58.675502    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:30:58.804665 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:58.804630    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:30:58.807365 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:30:58.807338    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7955e3e8_d873_4c53_ac2b_96d59a1709f6.slice/crio-3ec5192ff67b6c4d8973b1f7688dc3a09ba841ceb5e5a7d85bd173f02d3e7159 WatchSource:0}: Error finding container 3ec5192ff67b6c4d8973b1f7688dc3a09ba841ceb5e5a7d85bd173f02d3e7159: Status 404 returned error can't find the container with id 3ec5192ff67b6c4d8973b1f7688dc3a09ba841ceb5e5a7d85bd173f02d3e7159
Jul 28 11:30:59.670325 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:59.670284    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerStarted","Data":"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"}
Jul 28 11:30:59.670502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:30:59.670353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerStarted","Data":"3ec5192ff67b6c4d8973b1f7688dc3a09ba841ceb5e5a7d85bd173f02d3e7159"}
Jul 28 11:31:03.688024 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:03.687984    2569 generic.go:356] "Generic (PLEG): container finished" podID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerID="fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1" exitCode=0
Jul 28 11:31:03.688433 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:03.688057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerDied","Data":"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"}
Jul 28 11:31:04.694472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:04.694430    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerStarted","Data":"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"}
Jul 28 11:31:04.694886 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:04.694514    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerStarted","Data":"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"}
Jul 28 11:31:04.694886 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:04.694743    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:31:04.715410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:04.715353    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podStartSLOduration=3.828464518 podStartE2EDuration="7.71533802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.886873502s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:04.714608471 +0000 UTC m=+5022.635312523" watchObservedRunningTime="2026-07-28 11:31:04.71533802 +0000 UTC m=+5022.636042071"
Jul 28 11:31:05.699307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:05.699270    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:31:11.709136 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:11.709100    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:31:41.793861 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:41.793807    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 11:31:51.712000 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:51.711967    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:31:57.962344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:57.962305    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:31:57.962885 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:57.962550    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container" containerID="cri-o://98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec" gracePeriod=30
Jul 28 11:31:57.962885 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:57.962600    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kube-rbac-proxy" containerID="cri-o://83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40" gracePeriod=30
Jul 28 11:31:58.911052 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:58.911015    2569 generic.go:356] "Generic (PLEG): container finished" podID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerID="83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40" exitCode=2
Jul 28 11:31:58.911232 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:31:58.911082    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerDied","Data":"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"}
Jul 28 11:32:01.704018 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:01.703975    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.76:8643/healthz\": dial tcp 10.134.0.76:8643: connect: connection refused"
Jul 28 11:32:01.709620 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:01.709585    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.76:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.76:8080: connect: connection refused"
Jul 28 11:32:03.072796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.072757    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:32:03.076204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.076188    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.083831 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.083813    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 11:32:03.084555 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.084538    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 11:32:03.096100 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.096073    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:32:03.200322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.200290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.200322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.200332    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.200600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.200356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.200600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.200486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95v2s\" (UniqueName: \"kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.200600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.200533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.301619 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.301582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95v2s\" (UniqueName: \"kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.301619 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.301623    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.301883 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.301650    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.301883 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.301766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.301883 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.301824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.302198 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:32:03.301968    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 11:32:03.302198 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:32:03.302019    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls podName:710d9da2-c0dd-4bdf-9c22-783353bbd3b8 nodeName:}" failed. No retries permitted until 2026-07-28 11:32:03.80200326 +0000 UTC m=+5081.722707290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-h4r7k" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 11:32:03.302198 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.302169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.302414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.302396    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.302478 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.302437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.317249 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.317229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95v2s\" (UniqueName: \"kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.806548 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.806516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.808863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.808845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-h4r7k\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:03.985316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:03.985276    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:04.132109 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:04.132071    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:32:04.137242 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:32:04.137211    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710d9da2_c0dd_4bdf_9c22_783353bbd3b8.slice/crio-a533dd3fc3947e977eed183d5c424aa018e507120ddc5ac77a0c8b6178e6bb12 WatchSource:0}: Error finding container a533dd3fc3947e977eed183d5c424aa018e507120ddc5ac77a0c8b6178e6bb12: Status 404 returned error can't find the container with id a533dd3fc3947e977eed183d5c424aa018e507120ddc5ac77a0c8b6178e6bb12
Jul 28 11:32:04.938179 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:04.938148    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerStarted","Data":"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"}
Jul 28 11:32:04.938369 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:04.938200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerStarted","Data":"a533dd3fc3947e977eed183d5c424aa018e507120ddc5ac77a0c8b6178e6bb12"}
Jul 28 11:32:05.507621 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.507598    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:32:05.621278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg5x9\" (UniqueName: \"kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9\") pod \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") "
Jul 28 11:32:05.621278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert\") pod \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") "
Jul 28 11:32:05.621278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621248    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location\") pod \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") "
Jul 28 11:32:05.621278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621268    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") pod \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") "
Jul 28 11:32:05.621626 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config\") pod \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\" (UID: \"7955e3e8-d873-4c53-ac2b-96d59a1709f6\") "
Jul 28 11:32:05.621708 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621592    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7955e3e8-d873-4c53-ac2b-96d59a1709f6" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:32:05.621822 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621801    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7955e3e8-d873-4c53-ac2b-96d59a1709f6" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:32:05.621884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.621798    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7955e3e8-d873-4c53-ac2b-96d59a1709f6" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:32:05.623417 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.623394    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7955e3e8-d873-4c53-ac2b-96d59a1709f6" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:32:05.623518 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.623431    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9" (OuterVolumeSpecName: "kube-api-access-gg5x9") pod "7955e3e8-d873-4c53-ac2b-96d59a1709f6" (UID: "7955e3e8-d873-4c53-ac2b-96d59a1709f6"). InnerVolumeSpecName "kube-api-access-gg5x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:32:05.722111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.722076    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5x9\" (UniqueName: \"kubernetes.io/projected/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-api-access-gg5x9\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:32:05.722111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.722108    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:32:05.722305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.722123    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:32:05.722305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.722137    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7955e3e8-d873-4c53-ac2b-96d59a1709f6-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:32:05.722305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.722150    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7955e3e8-d873-4c53-ac2b-96d59a1709f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:32:05.943674 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.943639    2569 generic.go:356] "Generic (PLEG): container finished" podID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerID="98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec" exitCode=0
Jul 28 11:32:05.943868 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.943713    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"
Jul 28 11:32:05.943868 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.943727    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerDied","Data":"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"}
Jul 28 11:32:05.943868 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.943796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26" event={"ID":"7955e3e8-d873-4c53-ac2b-96d59a1709f6","Type":"ContainerDied","Data":"3ec5192ff67b6c4d8973b1f7688dc3a09ba841ceb5e5a7d85bd173f02d3e7159"}
Jul 28 11:32:05.943868 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.943830    2569 scope.go:117] "RemoveContainer" containerID="83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"
Jul 28 11:32:05.952884 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.952861    2569 scope.go:117] "RemoveContainer" containerID="98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"
Jul 28 11:32:05.962337 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.962321    2569 scope.go:117] "RemoveContainer" containerID="fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"
Jul 28 11:32:05.969472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.969448    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:32:05.969729 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.969716    2569 scope.go:117] "RemoveContainer" containerID="83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"
Jul 28 11:32:05.969984 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:32:05.969965    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40\": container with ID starting with 83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40 not found: ID does not exist" containerID="83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"
Jul 28 11:32:05.970041 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.969996    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40"} err="failed to get container status \"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40\": rpc error: code = NotFound desc = could not find container \"83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40\": container with ID starting with 83dee83e19756d98a4e068e6da0cc30a4cdc7f4742f30728bbd9e99499a63e40 not found: ID does not exist"
Jul 28 11:32:05.970041 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.970013    2569 scope.go:117] "RemoveContainer" containerID="98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"
Jul 28 11:32:05.970243 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:32:05.970227    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec\": container with ID starting with 98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec not found: ID does not exist" containerID="98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"
Jul 28 11:32:05.970288 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.970247    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec"} err="failed to get container status \"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec\": rpc error: code = NotFound desc = could not find container \"98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec\": container with ID starting with 98387e11781108a142ed05a389b58cfce0aab1313b30a2aec0f4ca211ded2eec not found: ID does not exist"
Jul 28 11:32:05.970288 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.970259    2569 scope.go:117] "RemoveContainer" containerID="fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"
Jul 28 11:32:05.970470 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:32:05.970454    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1\": container with ID starting with fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1 not found: ID does not exist" containerID="fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"
Jul 28 11:32:05.970509 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.970475    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1"} err="failed to get container status \"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1\": rpc error: code = NotFound desc = could not find container \"fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1\": container with ID starting with fe493d833a401a22794a703c4910e6a1d20cc9274262bf3db0484148778089e1 not found: ID does not exist"
Jul 28 11:32:05.973923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:05.973904    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-47x26"]
Jul 28 11:32:06.566145 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:06.566114    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" path="/var/lib/kubelet/pods/7955e3e8-d873-4c53-ac2b-96d59a1709f6/volumes"
Jul 28 11:32:08.958366 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:08.958327    2569 generic.go:356] "Generic (PLEG): container finished" podID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerID="7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a" exitCode=0
Jul 28 11:32:08.958904 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:08.958375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerDied","Data":"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"}
Jul 28 11:32:09.964764 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:09.964731    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerStarted","Data":"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"}
Jul 28 11:32:09.965165 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:09.964792    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerStarted","Data":"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"}
Jul 28 11:32:09.965165 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:09.964974    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:09.988082 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:09.988029    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podStartSLOduration=3.109027013 podStartE2EDuration="6.98801414s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878987127s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:32:09.987541967 +0000 UTC m=+5087.908246019" watchObservedRunningTime="2026-07-28 11:32:09.98801414 +0000 UTC m=+5087.908718191"
Jul 28 11:32:10.969365 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:10.969330    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:10.970558 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:10.970528    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:11.973800 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:11.973765    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:16.979610 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:16.979582    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:32:16.980110 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:16.980079    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:22.836483 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:22.836436    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:32:22.849965 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:22.849941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:32:26.980204 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:26.980162    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:36.980960 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:36.980919    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:46.980239 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:46.980147    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:32:56.980470 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:32:56.980429    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:33:06.981047 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:06.981003    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.77:8080: connect: connection refused"
Jul 28 11:33:16.981380 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:16.981351    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:33:23.123893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:23.123854    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:33:23.124312 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:23.124126    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container" containerID="cri-o://f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5" gracePeriod=30
Jul 28 11:33:23.124312 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:23.124189    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kube-rbac-proxy" containerID="cri-o://33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219" gracePeriod=30
Jul 28 11:33:23.266819 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:23.266782    2569 generic.go:356] "Generic (PLEG): container finished" podID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerID="33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219" exitCode=2
Jul 28 11:33:23.266971 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:23.266857    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerDied","Data":"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"}
Jul 28 11:33:26.569450 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.569426    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:33:26.702923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.702836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") pod \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") "
Jul 28 11:33:26.702923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.702880    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location\") pod \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") "
Jul 28 11:33:26.702923 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.702908    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert\") pod \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") "
Jul 28 11:33:26.703213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703020    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config\") pod \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") "
Jul 28 11:33:26.703213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703067    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95v2s\" (UniqueName: \"kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s\") pod \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\" (UID: \"710d9da2-c0dd-4bdf-9c22-783353bbd3b8\") "
Jul 28 11:33:26.703321 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703244    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "710d9da2-c0dd-4bdf-9c22-783353bbd3b8" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:33:26.703321 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703268    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "710d9da2-c0dd-4bdf-9c22-783353bbd3b8" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:33:26.703428 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703338    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:33:26.703428 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703353    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:33:26.703428 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.703382    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "710d9da2-c0dd-4bdf-9c22-783353bbd3b8" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:33:26.705138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.705113    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s" (OuterVolumeSpecName: "kube-api-access-95v2s") pod "710d9da2-c0dd-4bdf-9c22-783353bbd3b8" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8"). InnerVolumeSpecName "kube-api-access-95v2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:33:26.705138 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.705123    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "710d9da2-c0dd-4bdf-9c22-783353bbd3b8" (UID: "710d9da2-c0dd-4bdf-9c22-783353bbd3b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:33:26.804151 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.804117    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:33:26.804151 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.804146    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95v2s\" (UniqueName: \"kubernetes.io/projected/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-kube-api-access-95v2s\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:33:26.804151 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:26.804157    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/710d9da2-c0dd-4bdf-9c22-783353bbd3b8-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:33:27.284479 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.284443    2569 generic.go:356] "Generic (PLEG): container finished" podID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerID="f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5" exitCode=0
Jul 28 11:33:27.284707 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.284500    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"
Jul 28 11:33:27.284707 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.284516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerDied","Data":"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"}
Jul 28 11:33:27.284707 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.284580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k" event={"ID":"710d9da2-c0dd-4bdf-9c22-783353bbd3b8","Type":"ContainerDied","Data":"a533dd3fc3947e977eed183d5c424aa018e507120ddc5ac77a0c8b6178e6bb12"}
Jul 28 11:33:27.284707 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.284616    2569 scope.go:117] "RemoveContainer" containerID="33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"
Jul 28 11:33:27.293568 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.293549    2569 scope.go:117] "RemoveContainer" containerID="f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"
Jul 28 11:33:27.302887 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.302756    2569 scope.go:117] "RemoveContainer" containerID="7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"
Jul 28 11:33:27.307324 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.307300    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:33:27.310476 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.310455    2569 scope.go:117] "RemoveContainer" containerID="33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"
Jul 28 11:33:27.310793 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:33:27.310766    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219\": container with ID starting with 33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219 not found: ID does not exist" containerID="33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"
Jul 28 11:33:27.310913 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.310796    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219"} err="failed to get container status \"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219\": rpc error: code = NotFound desc = could not find container \"33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219\": container with ID starting with 33953b25d9f27439ce381477249927389f8ddbc9fcd88054ef7785956625a219 not found: ID does not exist"
Jul 28 11:33:27.310913 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.310816    2569 scope.go:117] "RemoveContainer" containerID="f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"
Jul 28 11:33:27.311047 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:33:27.311028    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5\": container with ID starting with f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5 not found: ID does not exist" containerID="f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"
Jul 28 11:33:27.311092 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.311053    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5"} err="failed to get container status \"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5\": rpc error: code = NotFound desc = could not find container \"f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5\": container with ID starting with f255b407bbcc2ed8a7593c72007633a5cd3fade659f3ebd537d187084e8312a5 not found: ID does not exist"
Jul 28 11:33:27.311092 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.311070    2569 scope.go:117] "RemoveContainer" containerID="7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"
Jul 28 11:33:27.311271 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:33:27.311244    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a\": container with ID starting with 7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a not found: ID does not exist" containerID="7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"
Jul 28 11:33:27.311357 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.311275    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a"} err="failed to get container status \"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a\": rpc error: code = NotFound desc = could not find container \"7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a\": container with ID starting with 7c5d927564f3df4921001b2ea3758ddfe3bf1f843d4fe5261b2b2c5c8587fc9a not found: ID does not exist"
Jul 28 11:33:27.312152 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:27.312133    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-h4r7k"]
Jul 28 11:33:28.289342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289312    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289679    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289710    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289720    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="storage-initializer"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289725    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="storage-initializer"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289733    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kube-rbac-proxy"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289738    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kube-rbac-proxy"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289747    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="storage-initializer"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289752    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="storage-initializer"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289760    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289767    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kube-rbac-proxy"
Jul 28 11:33:28.289804 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289785    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kube-rbac-proxy"
Jul 28 11:33:28.290186 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289835    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kserve-container"
Jul 28 11:33:28.290186 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289844    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kserve-container"
Jul 28 11:33:28.290186 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289854    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7955e3e8-d873-4c53-ac2b-96d59a1709f6" containerName="kube-rbac-proxy"
Jul 28 11:33:28.290186 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.289863    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" containerName="kube-rbac-proxy"
Jul 28 11:33:28.294414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.294395    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.297672 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.297653    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:33:28.297846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.297825    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:33:28.298847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.298828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 11:33:28.298946 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.298866    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 11:33:28.298946 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.298884    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 11:33:28.298946 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.298890    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:33:28.305907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.305889    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:33:28.417344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.417307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.417344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.417343    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.417569 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.417363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.417569 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.417479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pdv\" (UniqueName: \"kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.518823 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.518787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pdv\" (UniqueName: \"kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.519014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.518838    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.519014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.518856    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.519014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.518874    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.519322 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.519299    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.519504 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.519487    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.521272 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.521254    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.526979 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.526960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pdv\" (UniqueName: \"kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv\") pod \"isvc-sklearn-s3-predictor-7c588bddd4-5clsx\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.565961 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.565886    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="710d9da2-c0dd-4bdf-9c22-783353bbd3b8" path="/var/lib/kubelet/pods/710d9da2-c0dd-4bdf-9c22-783353bbd3b8/volumes"
Jul 28 11:33:28.606194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.606166    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:28.737911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:28.737874    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:33:29.295510 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:29.295465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerStarted","Data":"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"}
Jul 28 11:33:29.295916 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:29.295587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerStarted","Data":"1565a1b87663b6a443b056d28eaea5581374ee70936e37a27b839842d7d2e1e5"}
Jul 28 11:33:30.301126 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:30.301093    2569 generic.go:356] "Generic (PLEG): container finished" podID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerID="722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7" exitCode=0
Jul 28 11:33:30.301126 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:30.301130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerDied","Data":"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"}
Jul 28 11:33:31.306878 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:31.306840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerStarted","Data":"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"}
Jul 28 11:33:31.307314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:31.306901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerStarted","Data":"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"}
Jul 28 11:33:31.307314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:31.307052    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:31.327611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:31.327560    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podStartSLOduration=2.272023791 podStartE2EDuration="3.327545021s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.05552123s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:33:31.325762137 +0000 UTC m=+5169.246466189" watchObservedRunningTime="2026-07-28 11:33:31.327545021 +0000 UTC m=+5169.248249073"
Jul 28 11:33:32.311117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:32.311088    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:32.312393 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:32.312366    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:33:33.315458 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:33.315419    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:33:38.320417 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:38.320387    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:33:38.321069 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:38.321043    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:33:48.321877 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:48.321839    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:33:58.320934 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:33:58.320896    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:34:08.321346 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:08.321301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:34:18.320989 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:18.320904    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:34:28.320920 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:28.320877    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:34:38.321534 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:38.321495    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.78:8080: connect: connection refused"
Jul 28 11:34:48.321735 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:48.321680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:34:58.465552 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.465515    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:34:58.466014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.465828    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container" containerID="cri-o://35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f" gracePeriod=30
Jul 28 11:34:58.466014 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.465878    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kube-rbac-proxy" containerID="cri-o://d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f" gracePeriod=30
Jul 28 11:34:58.656967 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.656931    2569 generic.go:356] "Generic (PLEG): container finished" podID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerID="d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f" exitCode=2
Jul 28 11:34:58.657128 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.656978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerDied","Data":"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"}
Jul 28 11:34:58.677801 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.677764    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:34:58.681171 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.681155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.683736 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.683714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 11:34:58.683847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.683754    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 11:34:58.683847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.683767    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:34:58.691157 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.691132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:34:58.763462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.763426    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.763642 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.763472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nqr\" (UniqueName: \"kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.763642 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.763544    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.763642 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.763607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.763642 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.763627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.863999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.863957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.863999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.863996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nqr\" (UniqueName: \"kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864210 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864210 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:34:58.864189    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 11:34:58.864382 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:34:58.864265    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls podName:29939dbf-fe41-4b81-83d3-d126e93e600b nodeName:}" failed. No retries permitted until 2026-07-28 11:34:59.364244562 +0000 UTC m=+5257.284948595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 11:34:58.864434 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864414    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.864887 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.864858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:58.876111 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:58.876087    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nqr\" (UniqueName: \"kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:59.370013 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:59.369972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:59.372317 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:59.372298    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:59.591911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:59.591867    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:34:59.730098 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:34:59.730072    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:34:59.732789 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:34:59.732761    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29939dbf_fe41_4b81_83d3_d126e93e600b.slice/crio-980e9788e7a3699a4fb3e2a5c021be43266ad712b30eb849b2fabbfc345e81ca WatchSource:0}: Error finding container 980e9788e7a3699a4fb3e2a5c021be43266ad712b30eb849b2fabbfc345e81ca: Status 404 returned error can't find the container with id 980e9788e7a3699a4fb3e2a5c021be43266ad712b30eb849b2fabbfc345e81ca
Jul 28 11:35:00.667144 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:00.667097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerStarted","Data":"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"}
Jul 28 11:35:00.667537 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:00.667181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerStarted","Data":"980e9788e7a3699a4fb3e2a5c021be43266ad712b30eb849b2fabbfc345e81ca"}
Jul 28 11:35:01.672989 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:01.672954    2569 generic.go:356] "Generic (PLEG): container finished" podID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerID="4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad" exitCode=0
Jul 28 11:35:01.673376 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:01.673002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerDied","Data":"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"}
Jul 28 11:35:02.679299 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:02.679261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerStarted","Data":"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"}
Jul 28 11:35:02.679722 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:02.679356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerStarted","Data":"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"}
Jul 28 11:35:02.679722 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:02.679402    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:35:02.702254 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:02.702198    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podStartSLOduration=3.6664687320000002 podStartE2EDuration="4.702177317s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.035708585s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:02.698781704 +0000 UTC m=+5260.619485757" watchObservedRunningTime="2026-07-28 11:35:02.702177317 +0000 UTC m=+5260.622881372"
Jul 28 11:35:03.304591 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.304563    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:35:03.407049 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls\") pod \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") "
Jul 28 11:35:03.407237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407063    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24pdv\" (UniqueName: \"kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv\") pod \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") "
Jul 28 11:35:03.407237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407091    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config\") pod \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") "
Jul 28 11:35:03.407237 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407115    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location\") pod \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\" (UID: \"42ab5953-2f94-4ed6-9d88-535a44ee45ac\") "
Jul 28 11:35:03.407462 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407435    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42ab5953-2f94-4ed6-9d88-535a44ee45ac" (UID: "42ab5953-2f94-4ed6-9d88-535a44ee45ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:35:03.407529 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.407440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42ab5953-2f94-4ed6-9d88-535a44ee45ac" (UID: "42ab5953-2f94-4ed6-9d88-535a44ee45ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:35:03.409104 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.409080    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv" (OuterVolumeSpecName: "kube-api-access-24pdv") pod "42ab5953-2f94-4ed6-9d88-535a44ee45ac" (UID: "42ab5953-2f94-4ed6-9d88-535a44ee45ac"). InnerVolumeSpecName "kube-api-access-24pdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:35:03.409104 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.409089    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42ab5953-2f94-4ed6-9d88-535a44ee45ac" (UID: "42ab5953-2f94-4ed6-9d88-535a44ee45ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:35:03.508464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.508428    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ab5953-2f94-4ed6-9d88-535a44ee45ac-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:35:03.508464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.508457    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pdv\" (UniqueName: \"kubernetes.io/projected/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-api-access-24pdv\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:35:03.508464 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.508467    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:35:03.508728 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.508476    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ab5953-2f94-4ed6-9d88-535a44ee45ac-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:35:03.685502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685411    2569 generic.go:356] "Generic (PLEG): container finished" podID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerID="35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f" exitCode=0
Jul 28 11:35:03.685502 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685470    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"
Jul 28 11:35:03.685982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerDied","Data":"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"}
Jul 28 11:35:03.685982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx" event={"ID":"42ab5953-2f94-4ed6-9d88-535a44ee45ac","Type":"ContainerDied","Data":"1565a1b87663b6a443b056d28eaea5581374ee70936e37a27b839842d7d2e1e5"}
Jul 28 11:35:03.685982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685601    2569 scope.go:117] "RemoveContainer" containerID="d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"
Jul 28 11:35:03.685982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.685969    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:35:03.687338 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.687312    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:03.694172 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.694155    2569 scope.go:117] "RemoveContainer" containerID="35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"
Jul 28 11:35:03.701724 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.701706    2569 scope.go:117] "RemoveContainer" containerID="722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"
Jul 28 11:35:03.708765 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.708750    2569 scope.go:117] "RemoveContainer" containerID="d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"
Jul 28 11:35:03.709006 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:35:03.708987    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f\": container with ID starting with d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f not found: ID does not exist" containerID="d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"
Jul 28 11:35:03.709054 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.709014    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f"} err="failed to get container status \"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f\": rpc error: code = NotFound desc = could not find container \"d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f\": container with ID starting with d55d96338b4c2d5925fc8bf5dddb20bcbf4089b57b28d0bbfdebe30fd02c428f not found: ID does not exist"
Jul 28 11:35:03.709054 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.709032    2569 scope.go:117] "RemoveContainer" containerID="35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"
Jul 28 11:35:03.709258 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:35:03.709240    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f\": container with ID starting with 35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f not found: ID does not exist" containerID="35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"
Jul 28 11:35:03.709305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.709266    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f"} err="failed to get container status \"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f\": rpc error: code = NotFound desc = could not find container \"35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f\": container with ID starting with 35c9d5af1fdfd2dc455c63ae601bf5a12a346726e669bb79cbdf24dbc3fda84f not found: ID does not exist"
Jul 28 11:35:03.709305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.709285    2569 scope.go:117] "RemoveContainer" containerID="722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"
Jul 28 11:35:03.709506 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:35:03.709486    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7\": container with ID starting with 722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7 not found: ID does not exist" containerID="722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"
Jul 28 11:35:03.709554 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.709513    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7"} err="failed to get container status \"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7\": rpc error: code = NotFound desc = could not find container \"722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7\": container with ID starting with 722bfd0d91808337bd5212eba129995a56e058612923ecf6e3f979cce4f8fce7 not found: ID does not exist"
Jul 28 11:35:03.712532 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.712510    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:35:03.718759 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:03.718737    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7c588bddd4-5clsx"]
Jul 28 11:35:04.566492 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:04.566458    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" path="/var/lib/kubelet/pods/42ab5953-2f94-4ed6-9d88-535a44ee45ac/volumes"
Jul 28 11:35:04.691044 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:04.691012    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:09.700738 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:09.700710    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:35:09.701366 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:09.701335    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:19.702277 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:19.702236    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:29.701336 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:29.701292    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:39.701647 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:39.701607    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:49.701996 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:49.701909    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:35:59.701564 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:35:59.701521    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:36:09.702341 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:09.702298    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:36:19.702338 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:19.702300    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:36:28.620791 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:28.620747    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:36:28.621259 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:28.621038    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" containerID="cri-o://71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a" gracePeriod=30
Jul 28 11:36:28.621259 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:28.621129    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kube-rbac-proxy" containerID="cri-o://35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6" gracePeriod=30
Jul 28 11:36:29.041500 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.041466    2569 generic.go:356] "Generic (PLEG): container finished" podID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerID="35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6" exitCode=2
Jul 28 11:36:29.041788 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.041538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerDied","Data":"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"}
Jul 28 11:36:29.691810 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.691763    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.79:8643/healthz\": dial tcp 10.134.0.79:8643: connect: connection refused"
Jul 28 11:36:29.702303 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.702267    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.79:8080: connect: connection refused"
Jul 28 11:36:29.753860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.753830    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:29.754142 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754130    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container"
Jul 28 11:36:29.754191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754143    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container"
Jul 28 11:36:29.754191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754156    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kube-rbac-proxy"
Jul 28 11:36:29.754191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754162    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kube-rbac-proxy"
Jul 28 11:36:29.754191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754174    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="storage-initializer"
Jul 28 11:36:29.754191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754180    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="storage-initializer"
Jul 28 11:36:29.754373 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754247    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kserve-container"
Jul 28 11:36:29.754373 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.754254    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ab5953-2f94-4ed6-9d88-535a44ee45ac" containerName="kube-rbac-proxy"
Jul 28 11:36:29.757197 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.757181    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.759900 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.759885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 11:36:29.759986 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.759907    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 11:36:29.772846 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.772815    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:29.825422 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.825389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.825611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.825447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.825611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.825503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24vb\" (UniqueName: \"kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.825611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.825543    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.926927 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.926892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.927124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.926944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.927124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.926971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24vb\" (UniqueName: \"kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.927124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.927000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.927386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.927367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.927703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.927661    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.929392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.929371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:29.937130 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:29.937102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24vb\" (UniqueName: \"kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:30.066763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:30.066645    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:30.197342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:30.197307    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:30.200299 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:36:30.200270    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934a9b38_f272_4e43_9f52_0b4594c94a42.slice/crio-247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94 WatchSource:0}: Error finding container 247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94: Status 404 returned error can't find the container with id 247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94
Jul 28 11:36:31.050998 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:31.050961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerStarted","Data":"0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"}
Jul 28 11:36:31.051400 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:31.051020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerStarted","Data":"247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94"}
Jul 28 11:36:33.264779 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.264756    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:36:33.354999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.354899    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location\") pod \"29939dbf-fe41-4b81-83d3-d126e93e600b\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") "
Jul 28 11:36:33.354999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.354942    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config\") pod \"29939dbf-fe41-4b81-83d3-d126e93e600b\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") "
Jul 28 11:36:33.355248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355022    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert\") pod \"29939dbf-fe41-4b81-83d3-d126e93e600b\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") "
Jul 28 11:36:33.355248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355058    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9nqr\" (UniqueName: \"kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr\") pod \"29939dbf-fe41-4b81-83d3-d126e93e600b\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") "
Jul 28 11:36:33.355248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") pod \"29939dbf-fe41-4b81-83d3-d126e93e600b\" (UID: \"29939dbf-fe41-4b81-83d3-d126e93e600b\") "
Jul 28 11:36:33.355378 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355278    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29939dbf-fe41-4b81-83d3-d126e93e600b" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:33.355378 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355354    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29939dbf-fe41-4b81-83d3-d126e93e600b" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:36:33.355528 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.355498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29939dbf-fe41-4b81-83d3-d126e93e600b" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:36:33.357242 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.357223    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29939dbf-fe41-4b81-83d3-d126e93e600b" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:36:33.357345 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.357238    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr" (OuterVolumeSpecName: "kube-api-access-s9nqr") pod "29939dbf-fe41-4b81-83d3-d126e93e600b" (UID: "29939dbf-fe41-4b81-83d3-d126e93e600b"). InnerVolumeSpecName "kube-api-access-s9nqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:36:33.455931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.455874    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:33.455931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.455925    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nqr\" (UniqueName: \"kubernetes.io/projected/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-api-access-s9nqr\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:33.455931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.455941    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29939dbf-fe41-4b81-83d3-d126e93e600b-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:33.456192 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.455954    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29939dbf-fe41-4b81-83d3-d126e93e600b-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:33.456192 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:33.455965    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29939dbf-fe41-4b81-83d3-d126e93e600b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:34.068369 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.068340    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/0.log"
Jul 28 11:36:34.068570 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.068385    2569 generic.go:356] "Generic (PLEG): container finished" podID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerID="0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5" exitCode=1
Jul 28 11:36:34.068570 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.068459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerDied","Data":"0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"}
Jul 28 11:36:34.070547 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.070524    2569 generic.go:356] "Generic (PLEG): container finished" podID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerID="71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a" exitCode=0
Jul 28 11:36:34.070703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.070557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerDied","Data":"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"}
Jul 28 11:36:34.070703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.070592    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"
Jul 28 11:36:34.070703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.070601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w" event={"ID":"29939dbf-fe41-4b81-83d3-d126e93e600b","Type":"ContainerDied","Data":"980e9788e7a3699a4fb3e2a5c021be43266ad712b30eb849b2fabbfc345e81ca"}
Jul 28 11:36:34.070703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.070626    2569 scope.go:117] "RemoveContainer" containerID="35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"
Jul 28 11:36:34.080235 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.080217    2569 scope.go:117] "RemoveContainer" containerID="71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"
Jul 28 11:36:34.087677 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.087652    2569 scope.go:117] "RemoveContainer" containerID="4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"
Jul 28 11:36:34.102490 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.102465    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:36:34.102569 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.102514    2569 scope.go:117] "RemoveContainer" containerID="35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"
Jul 28 11:36:34.102883 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:34.102857    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6\": container with ID starting with 35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6 not found: ID does not exist" containerID="35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"
Jul 28 11:36:34.102943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.102891    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6"} err="failed to get container status \"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6\": rpc error: code = NotFound desc = could not find container \"35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6\": container with ID starting with 35eca29241efae44c9b52885c18acc051f0da06509dc756bcd3b781cdb7044a6 not found: ID does not exist"
Jul 28 11:36:34.102943 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.102908    2569 scope.go:117] "RemoveContainer" containerID="71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"
Jul 28 11:36:34.103211 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:34.103187    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a\": container with ID starting with 71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a not found: ID does not exist" containerID="71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"
Jul 28 11:36:34.103264 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.103225    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a"} err="failed to get container status \"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a\": rpc error: code = NotFound desc = could not find container \"71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a\": container with ID starting with 71f73d30931ac4b3e2c506b3c2111d650d4a78bd429e52a8b2593150d332179a not found: ID does not exist"
Jul 28 11:36:34.103264 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.103248    2569 scope.go:117] "RemoveContainer" containerID="4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"
Jul 28 11:36:34.103489 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:34.103474    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad\": container with ID starting with 4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad not found: ID does not exist" containerID="4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"
Jul 28 11:36:34.103527 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.103495    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad"} err="failed to get container status \"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad\": rpc error: code = NotFound desc = could not find container \"4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad\": container with ID starting with 4125859b805a1c0e5d9b66b25ec64075a39a1c99cb1679be70fd119b4860cbad not found: ID does not exist"
Jul 28 11:36:34.107040 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.107015    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-97b95f984-g5t6w"]
Jul 28 11:36:34.566446 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:34.566406    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" path="/var/lib/kubelet/pods/29939dbf-fe41-4b81-83d3-d126e93e600b/volumes"
Jul 28 11:36:35.076617 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:35.076587    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/0.log"
Jul 28 11:36:35.076813 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:35.076661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerStarted","Data":"b51987ae60470bfc48abfe3cae019c456c7e95bd2b8690a222caba927e2fc353"}
Jul 28 11:36:37.085371 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.085294    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/1.log"
Jul 28 11:36:37.085746 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.085654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/0.log"
Jul 28 11:36:37.085746 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.085698    2569 generic.go:356] "Generic (PLEG): container finished" podID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerID="b51987ae60470bfc48abfe3cae019c456c7e95bd2b8690a222caba927e2fc353" exitCode=1
Jul 28 11:36:37.085746 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.085714    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerDied","Data":"b51987ae60470bfc48abfe3cae019c456c7e95bd2b8690a222caba927e2fc353"}
Jul 28 11:36:37.085863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.085777    2569 scope.go:117] "RemoveContainer" containerID="0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"
Jul 28 11:36:37.086133 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.086116    2569 scope.go:117] "RemoveContainer" containerID="0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"
Jul 28 11:36:37.096401 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:37.096373    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_kserve-ci-e2e-test(934a9b38-f272-4e43-9f52-0b4594c94a42)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42"
Jul 28 11:36:37.096517 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:37.096442    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_kserve-ci-e2e-test_934a9b38-f272-4e43-9f52-0b4594c94a42_0 in pod sandbox 247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94 from index: no such id: '0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5'" containerID="0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"
Jul 28 11:36:37.096517 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:37.096477    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_kserve-ci-e2e-test_934a9b38-f272-4e43-9f52-0b4594c94a42_0 in pod sandbox 247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94 from index: no such id: '0b4875f46061fe379c626a40eb85bdf52020ac3edc5c458184f8b2df0ad6d0a5'"
Jul 28 11:36:38.091656 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:38.091630    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/1.log"
Jul 28 11:36:39.718764 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.718727    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:39.856819 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.856795    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/1.log"
Jul 28 11:36:39.860592 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.860576    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:39.912922 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.912879    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls\") pod \"934a9b38-f272-4e43-9f52-0b4594c94a42\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") "
Jul 28 11:36:39.913131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.912949    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config\") pod \"934a9b38-f272-4e43-9f52-0b4594c94a42\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") "
Jul 28 11:36:39.913131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.913011    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x24vb\" (UniqueName: \"kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb\") pod \"934a9b38-f272-4e43-9f52-0b4594c94a42\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") "
Jul 28 11:36:39.913131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.913069    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location\") pod \"934a9b38-f272-4e43-9f52-0b4594c94a42\" (UID: \"934a9b38-f272-4e43-9f52-0b4594c94a42\") "
Jul 28 11:36:39.913414 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.913384    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "934a9b38-f272-4e43-9f52-0b4594c94a42" (UID: "934a9b38-f272-4e43-9f52-0b4594c94a42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:36:39.913521 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.913427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "934a9b38-f272-4e43-9f52-0b4594c94a42" (UID: "934a9b38-f272-4e43-9f52-0b4594c94a42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:36:39.915057 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.915034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "934a9b38-f272-4e43-9f52-0b4594c94a42" (UID: "934a9b38-f272-4e43-9f52-0b4594c94a42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:36:39.915156 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:39.915100    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb" (OuterVolumeSpecName: "kube-api-access-x24vb") pod "934a9b38-f272-4e43-9f52-0b4594c94a42" (UID: "934a9b38-f272-4e43-9f52-0b4594c94a42"). InnerVolumeSpecName "kube-api-access-x24vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:36:40.013737 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.013618    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/934a9b38-f272-4e43-9f52-0b4594c94a42-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:40.013737 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.013649    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/934a9b38-f272-4e43-9f52-0b4594c94a42-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:40.013737 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.013658    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:40.013737 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.013668    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24vb\" (UniqueName: \"kubernetes.io/projected/934a9b38-f272-4e43-9f52-0b4594c94a42-kube-api-access-x24vb\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:36:40.101418 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.101391    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx_934a9b38-f272-4e43-9f52-0b4594c94a42/storage-initializer/1.log"
Jul 28 11:36:40.101598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.101472    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx" event={"ID":"934a9b38-f272-4e43-9f52-0b4594c94a42","Type":"ContainerDied","Data":"247ada9a651ecb0cb4cfdd7bb07f1e8b65e2d6247fc157281f5a57a4a2ee5f94"}
Jul 28 11:36:40.101598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.101491    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"
Jul 28 11:36:40.101598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.101574    2569 scope.go:117] "RemoveContainer" containerID="b51987ae60470bfc48abfe3cae019c456c7e95bd2b8690a222caba927e2fc353"
Jul 28 11:36:40.137644 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.137613    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:40.139567 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.139540    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bb4c6dd88-ptzkx"]
Jul 28 11:36:40.566265 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.566230    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" path="/var/lib/kubelet/pods/934a9b38-f272-4e43-9f52-0b4594c94a42/volumes"
Jul 28 11:36:40.827286 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827191    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827634    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827653    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827665    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827673    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827702    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827712    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827722    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827729    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827755    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kube-rbac-proxy"
Jul 28 11:36:40.827781 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827762    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kube-rbac-proxy"
Jul 28 11:36:40.828302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827843    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kserve-container"
Jul 28 11:36:40.828302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827859    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="29939dbf-fe41-4b81-83d3-d126e93e600b" containerName="kube-rbac-proxy"
Jul 28 11:36:40.828302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827869    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.828302 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.827879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="934a9b38-f272-4e43-9f52-0b4594c94a42" containerName="storage-initializer"
Jul 28 11:36:40.832407 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.832383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:40.835107 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835082    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 11:36:40.835270 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835140    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tww8p\""
Jul 28 11:36:40.835342 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835327    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 11:36:40.835670 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835650    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 11:36:40.835870 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835856    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 11:36:40.835931 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.835888    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 11:36:40.836217 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.836202    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:36:40.841123 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.841092    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:36:40.919492 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.919443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbmz\" (UniqueName: \"kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:40.919663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.919504    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:40.919663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.919547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:40.919663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.919571    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:40.919663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:40.919615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020468 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020468 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020472    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020786 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020786 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbmz\" (UniqueName: \"kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020786 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.020786 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:41.020613    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 11:36:41.020786 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:36:41.020712    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls podName:c7fc9240-b38a-4eea-871b-88b286959012 nodeName:}" failed. No retries permitted until 2026-07-28 11:36:41.520664892 +0000 UTC m=+5359.441368936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" (UID: "c7fc9240-b38a-4eea-871b-88b286959012") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 11:36:41.021028 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.020994    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.021194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.021175    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.021236 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.021221    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.031494 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.031456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbmz\" (UniqueName: \"kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.524487 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.524444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.526835 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.526815    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.743967 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.743927    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:41.877443 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:41.877417    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:36:41.879972 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:36:41.879946    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fc9240_b38a_4eea_871b_88b286959012.slice/crio-31d6d3384feddd314aae999d1da02e7995bcd6ac636bd587b3ff0ea63ead014b WatchSource:0}: Error finding container 31d6d3384feddd314aae999d1da02e7995bcd6ac636bd587b3ff0ea63ead014b: Status 404 returned error can't find the container with id 31d6d3384feddd314aae999d1da02e7995bcd6ac636bd587b3ff0ea63ead014b
Jul 28 11:36:42.112278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:42.112198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerStarted","Data":"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"}
Jul 28 11:36:42.112278 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:42.112256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerStarted","Data":"31d6d3384feddd314aae999d1da02e7995bcd6ac636bd587b3ff0ea63ead014b"}
Jul 28 11:36:43.118226 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:43.118193    2569 generic.go:356] "Generic (PLEG): container finished" podID="c7fc9240-b38a-4eea-871b-88b286959012" containerID="f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea" exitCode=0
Jul 28 11:36:43.118609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:43.118265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerDied","Data":"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"}
Jul 28 11:36:44.124503 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.124471    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerStarted","Data":"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"}
Jul 28 11:36:44.124938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.124529    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerStarted","Data":"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"}
Jul 28 11:36:44.124938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.124554    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:44.124938 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.124647    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:44.125859 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.125837    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:36:44.145917 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:44.145864    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podStartSLOduration=3.112585548 podStartE2EDuration="4.145848634s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.033263086s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:36:44.144305555 +0000 UTC m=+5362.065009608" watchObservedRunningTime="2026-07-28 11:36:44.145848634 +0000 UTC m=+5362.066552686"
Jul 28 11:36:45.128585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:45.128549    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:36:50.133163 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:50.133126    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:36:50.133634 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:36:50.133606    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:00.134589 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:00.134543    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:10.134422 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:10.134381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:20.133609 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:20.133524    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:22.861375 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:22.861348    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:37:22.882870 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:22.882842    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:37:30.134493 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:30.134451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:40.134083 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:40.134045    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:37:50.133778 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:37:50.133739    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.81:8080: connect: connection refused"
Jul 28 11:38:00.134669 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:00.134629    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:38:00.883545 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:00.883500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:38:00.883768 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:00.883742    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container" containerID="cri-o://b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b" gracePeriod=30
Jul 28 11:38:00.883838 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:00.883809    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kube-rbac-proxy" containerID="cri-o://08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db" gracePeriod=30
Jul 28 11:38:01.443521 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.443488    2569 generic.go:356] "Generic (PLEG): container finished" podID="c7fc9240-b38a-4eea-871b-88b286959012" containerID="08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db" exitCode=2
Jul 28 11:38:01.443940 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.443561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerDied","Data":"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"}
Jul 28 11:38:01.955072 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.955031    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:01.958631 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.958606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:01.961465 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.961446    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 11:38:01.961721 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.961705    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 11:38:01.967726 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:01.967704    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:02.111011 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.110972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.111194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.111091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thh8x\" (UniqueName: \"kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.111194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.111128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.111194 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.111151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.212381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.212285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thh8x\" (UniqueName: \"kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.212381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.212327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.212381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.212354    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.212381 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.212383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.212873 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:02.212503    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 28 11:38:02.212873 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:02.212572    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls podName:126c85a2-37ba-46ab-8692-27d2378c0973 nodeName:}" failed. No retries permitted until 2026-07-28 11:38:02.71255081 +0000 UTC m=+5440.633254840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" (UID: "126c85a2-37ba-46ab-8692-27d2378c0973") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 28 11:38:02.212873 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.212797    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.213105 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.213085    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.220784 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.220763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thh8x\" (UniqueName: \"kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.717002 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.716964    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.719437 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.719409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:02.869182 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:02.869142    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:03.001028 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:03.000864    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:03.003906 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:38:03.003883    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126c85a2_37ba_46ab_8692_27d2378c0973.slice/crio-8c06f7d56939b20b5c171437f31f6242e9085931f5c7564c7ea9c14ce27efb2f WatchSource:0}: Error finding container 8c06f7d56939b20b5c171437f31f6242e9085931f5c7564c7ea9c14ce27efb2f: Status 404 returned error can't find the container with id 8c06f7d56939b20b5c171437f31f6242e9085931f5c7564c7ea9c14ce27efb2f
Jul 28 11:38:03.456842 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:03.456806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerStarted","Data":"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"}
Jul 28 11:38:03.457035 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:03.456867    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerStarted","Data":"8c06f7d56939b20b5c171437f31f6242e9085931f5c7564c7ea9c14ce27efb2f"}
Jul 28 11:38:05.128843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.128793    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.81:8643/healthz\": dial tcp 10.134.0.81:8643: connect: connection refused"
Jul 28 11:38:05.627483 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.627459    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:38:05.641540 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641518    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") pod \"c7fc9240-b38a-4eea-871b-88b286959012\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") "
Jul 28 11:38:05.641671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641547    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location\") pod \"c7fc9240-b38a-4eea-871b-88b286959012\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") "
Jul 28 11:38:05.641671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbmz\" (UniqueName: \"kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz\") pod \"c7fc9240-b38a-4eea-871b-88b286959012\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") "
Jul 28 11:38:05.641671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert\") pod \"c7fc9240-b38a-4eea-871b-88b286959012\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") "
Jul 28 11:38:05.641671 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641664    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config\") pod \"c7fc9240-b38a-4eea-871b-88b286959012\" (UID: \"c7fc9240-b38a-4eea-871b-88b286959012\") "
Jul 28 11:38:05.641999 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.641966    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7fc9240-b38a-4eea-871b-88b286959012" (UID: "c7fc9240-b38a-4eea-871b-88b286959012"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:38:05.642124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.642017    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7fc9240-b38a-4eea-871b-88b286959012" (UID: "c7fc9240-b38a-4eea-871b-88b286959012"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:38:05.642124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.642036    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7fc9240-b38a-4eea-871b-88b286959012" (UID: "c7fc9240-b38a-4eea-871b-88b286959012"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:38:05.643720 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.643673    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7fc9240-b38a-4eea-871b-88b286959012" (UID: "c7fc9240-b38a-4eea-871b-88b286959012"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:38:05.643720 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.643702    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz" (OuterVolumeSpecName: "kube-api-access-lkbmz") pod "c7fc9240-b38a-4eea-871b-88b286959012" (UID: "c7fc9240-b38a-4eea-871b-88b286959012"). InnerVolumeSpecName "kube-api-access-lkbmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:38:05.743096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.743005    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbmz\" (UniqueName: \"kubernetes.io/projected/c7fc9240-b38a-4eea-871b-88b286959012-kube-api-access-lkbmz\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:05.743096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.743037    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:05.743096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.743046    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7fc9240-b38a-4eea-871b-88b286959012-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:05.743096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.743055    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7fc9240-b38a-4eea-871b-88b286959012-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:05.743096 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:05.743063    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7fc9240-b38a-4eea-871b-88b286959012-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:06.472249 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.472213    2569 generic.go:356] "Generic (PLEG): container finished" podID="c7fc9240-b38a-4eea-871b-88b286959012" containerID="b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b" exitCode=0
Jul 28 11:38:06.472662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.472271    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"
Jul 28 11:38:06.472662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.472299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerDied","Data":"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"}
Jul 28 11:38:06.472662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.472387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w" event={"ID":"c7fc9240-b38a-4eea-871b-88b286959012","Type":"ContainerDied","Data":"31d6d3384feddd314aae999d1da02e7995bcd6ac636bd587b3ff0ea63ead014b"}
Jul 28 11:38:06.472662 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.472414    2569 scope.go:117] "RemoveContainer" containerID="08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"
Jul 28 11:38:06.482126 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.482075    2569 scope.go:117] "RemoveContainer" containerID="b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"
Jul 28 11:38:06.490169 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.490145    2569 scope.go:117] "RemoveContainer" containerID="f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"
Jul 28 11:38:06.496258 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.496225    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:38:06.498314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.498182    2569 scope.go:117] "RemoveContainer" containerID="08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"
Jul 28 11:38:06.498624 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:06.498595    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db\": container with ID starting with 08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db not found: ID does not exist" containerID="08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"
Jul 28 11:38:06.498768 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.498641    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db"} err="failed to get container status \"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db\": rpc error: code = NotFound desc = could not find container \"08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db\": container with ID starting with 08bf0091ab573947af904f190615663403d1a60afb1cb294590d7cdc61c787db not found: ID does not exist"
Jul 28 11:38:06.498768 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.498668    2569 scope.go:117] "RemoveContainer" containerID="b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"
Jul 28 11:38:06.499028 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:06.498996    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b\": container with ID starting with b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b not found: ID does not exist" containerID="b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"
Jul 28 11:38:06.499135 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.499030    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b"} err="failed to get container status \"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b\": rpc error: code = NotFound desc = could not find container \"b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b\": container with ID starting with b99f8cb9774e83bd8c80a4504419e495fd7339fed472b5735cf6d6aa3e725d4b not found: ID does not exist"
Jul 28 11:38:06.499135 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.499050    2569 scope.go:117] "RemoveContainer" containerID="f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"
Jul 28 11:38:06.499349 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:06.499328    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea\": container with ID starting with f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea not found: ID does not exist" containerID="f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"
Jul 28 11:38:06.499392 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.499354    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea"} err="failed to get container status \"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea\": rpc error: code = NotFound desc = could not find container \"f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea\": container with ID starting with f6d5ad4d9c24334c6d185c357c98507967a069b59fab1445d2f41f5662702cea not found: ID does not exist"
Jul 28 11:38:06.500314 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.500271    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-d7dd8d9b8-67z5w"]
Jul 28 11:38:06.566318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:06.566289    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7fc9240-b38a-4eea-871b-88b286959012" path="/var/lib/kubelet/pods/c7fc9240-b38a-4eea-871b-88b286959012/volumes"
Jul 28 11:38:08.483180 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:08.483151    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/0.log"
Jul 28 11:38:08.483613 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:08.483196    2569 generic.go:356] "Generic (PLEG): container finished" podID="126c85a2-37ba-46ab-8692-27d2378c0973" containerID="3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2" exitCode=1
Jul 28 11:38:08.483613 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:08.483276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerDied","Data":"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"}
Jul 28 11:38:09.488853 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:09.488824    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/0.log"
Jul 28 11:38:09.489235 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:09.488911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerStarted","Data":"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"}
Jul 28 11:38:11.955600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:11.955560    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:11.956172 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:11.955798    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer" containerID="cri-o://506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c" gracePeriod=30
Jul 28 11:38:13.067408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067363    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067707    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kube-rbac-proxy"
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067723    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kube-rbac-proxy"
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067739    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container"
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067744    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container"
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067761    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="storage-initializer"
Jul 28 11:38:13.067806 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067768    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="storage-initializer"
Jul 28 11:38:13.068005 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067820    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kserve-container"
Jul 28 11:38:13.068005 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.067828    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7fc9240-b38a-4eea-871b-88b286959012" containerName="kube-rbac-proxy"
Jul 28 11:38:13.071042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.071024    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.074246 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.074228    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 11:38:13.074382 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.074228    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 11:38:13.074382 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.074228    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 11:38:13.083316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.083286    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:38:13.102313 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.102288    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/1.log"
Jul 28 11:38:13.102716 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.102680    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/0.log"
Jul 28 11:38:13.104982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.104963    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:13.104982 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.104978    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.105131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.105005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.105131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.105037    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2v8s\" (UniqueName: \"kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.105131 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.105088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.105250 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.105148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.205710 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205580    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config\") pod \"126c85a2-37ba-46ab-8692-27d2378c0973\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") "
Jul 28 11:38:13.205710 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205676    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thh8x\" (UniqueName: \"kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x\") pod \"126c85a2-37ba-46ab-8692-27d2378c0973\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") "
Jul 28 11:38:13.205907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205739    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location\") pod \"126c85a2-37ba-46ab-8692-27d2378c0973\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") "
Jul 28 11:38:13.205907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205768    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") pod \"126c85a2-37ba-46ab-8692-27d2378c0973\" (UID: \"126c85a2-37ba-46ab-8692-27d2378c0973\") "
Jul 28 11:38:13.205907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.205907 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205888    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2v8s\" (UniqueName: \"kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.205975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "126c85a2-37ba-46ab-8692-27d2378c0973" (UID: "126c85a2-37ba-46ab-8692-27d2378c0973"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:38:13.206102 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:13.206053    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 11:38:13.206328 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206096    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206328 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:13.206152    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls podName:9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8 nodeName:}" failed. No retries permitted until 2026-07-28 11:38:13.706128879 +0000 UTC m=+5451.626832908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 11:38:13.206328 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206175    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/126c85a2-37ba-46ab-8692-27d2378c0973-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:13.206515 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206493    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206644    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.206757 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206713    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "126c85a2-37ba-46ab-8692-27d2378c0973" (UID: "126c85a2-37ba-46ab-8692-27d2378c0973"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:38:13.206800 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.206757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.207901 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.207878    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "126c85a2-37ba-46ab-8692-27d2378c0973" (UID: "126c85a2-37ba-46ab-8692-27d2378c0973"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:38:13.208015 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.208000    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x" (OuterVolumeSpecName: "kube-api-access-thh8x") pod "126c85a2-37ba-46ab-8692-27d2378c0973" (UID: "126c85a2-37ba-46ab-8692-27d2378c0973"). InnerVolumeSpecName "kube-api-access-thh8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:38:13.214908 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.214890    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2v8s\" (UniqueName: \"kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.307455 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.307420    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thh8x\" (UniqueName: \"kubernetes.io/projected/126c85a2-37ba-46ab-8692-27d2378c0973-kube-api-access-thh8x\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:13.307455 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.307448    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/126c85a2-37ba-46ab-8692-27d2378c0973-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:13.307455 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.307458    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/126c85a2-37ba-46ab-8692-27d2378c0973-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:38:13.507518 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507439    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/1.log"
Jul 28 11:38:13.507813 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507798    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb_126c85a2-37ba-46ab-8692-27d2378c0973/storage-initializer/0.log"
Jul 28 11:38:13.507872 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507833    2569 generic.go:356] "Generic (PLEG): container finished" podID="126c85a2-37ba-46ab-8692-27d2378c0973" containerID="506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c" exitCode=1
Jul 28 11:38:13.507911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerDied","Data":"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"}
Jul 28 11:38:13.507911 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507904    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"
Jul 28 11:38:13.507975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507939    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb" event={"ID":"126c85a2-37ba-46ab-8692-27d2378c0973","Type":"ContainerDied","Data":"8c06f7d56939b20b5c171437f31f6242e9085931f5c7564c7ea9c14ce27efb2f"}
Jul 28 11:38:13.508010 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.507977    2569 scope.go:117] "RemoveContainer" containerID="506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"
Jul 28 11:38:13.516288 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.516270    2569 scope.go:117] "RemoveContainer" containerID="3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"
Jul 28 11:38:13.523581 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.523564    2569 scope.go:117] "RemoveContainer" containerID="506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"
Jul 28 11:38:13.523839 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:13.523819    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c\": container with ID starting with 506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c not found: ID does not exist" containerID="506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"
Jul 28 11:38:13.523882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.523848    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c"} err="failed to get container status \"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c\": rpc error: code = NotFound desc = could not find container \"506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c\": container with ID starting with 506a93e9461820a83d4517bfb5f798419faf938b11a5d7a0b3ca2f8c462cb83c not found: ID does not exist"
Jul 28 11:38:13.523882 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.523863    2569 scope.go:117] "RemoveContainer" containerID="3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"
Jul 28 11:38:13.524090 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:38:13.524072    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2\": container with ID starting with 3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2 not found: ID does not exist" containerID="3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"
Jul 28 11:38:13.524132 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.524097    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2"} err="failed to get container status \"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2\": rpc error: code = NotFound desc = could not find container \"3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2\": container with ID starting with 3a197b66f6b1b2bc2ada924578ede961ef93c43f61c7902a01d3c6d9d5cf94f2 not found: ID does not exist"
Jul 28 11:38:13.550236 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.550205    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:13.554125 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.554104    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-959565c6d-sc4cb"]
Jul 28 11:38:13.710756 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.710718    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.713117 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.713089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:13.983980 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:13.983942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:14.118977 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:14.118947    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:38:14.122568 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:38:14.122540    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1cc2b1_3acd_48dd_8da9_64b0a3bb70a8.slice/crio-2ee81fb6a264fcee149fc93ef381d63000611643bb4509dd3f92523be8257b2c WatchSource:0}: Error finding container 2ee81fb6a264fcee149fc93ef381d63000611643bb4509dd3f92523be8257b2c: Status 404 returned error can't find the container with id 2ee81fb6a264fcee149fc93ef381d63000611643bb4509dd3f92523be8257b2c
Jul 28 11:38:14.514511 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:14.514473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerStarted","Data":"cd5335af5f4af1500e1cbef89545010ccc9ea663214a2bcc9b28d3799a44e1c1"}
Jul 28 11:38:14.514711 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:14.514561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerStarted","Data":"2ee81fb6a264fcee149fc93ef381d63000611643bb4509dd3f92523be8257b2c"}
Jul 28 11:38:14.566843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:14.566805    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" path="/var/lib/kubelet/pods/126c85a2-37ba-46ab-8692-27d2378c0973/volumes"
Jul 28 11:38:15.519789 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:15.519679    2569 generic.go:356] "Generic (PLEG): container finished" podID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerID="cd5335af5f4af1500e1cbef89545010ccc9ea663214a2bcc9b28d3799a44e1c1" exitCode=0
Jul 28 11:38:15.520167 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:15.519778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerDied","Data":"cd5335af5f4af1500e1cbef89545010ccc9ea663214a2bcc9b28d3799a44e1c1"}
Jul 28 11:38:16.526160 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.526122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerStarted","Data":"209bf9a419e3e6e4c57647449ba8267542123c431772f271ead3cc04ecb54487"}
Jul 28 11:38:16.526655 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.526188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerStarted","Data":"102e754f697d8a5d8471c2fd9b4aec7d1ca43209d02c61509b9a1677d12776b2"}
Jul 28 11:38:16.526655 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.526224    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:16.526655 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.526252    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:16.527570 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.527542    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:38:16.547549 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:16.547500    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podStartSLOduration=2.522508631 podStartE2EDuration="3.547483732s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.024975101s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:38:16.546193139 +0000 UTC m=+5454.466897191" watchObservedRunningTime="2026-07-28 11:38:16.547483732 +0000 UTC m=+5454.468187784"
Jul 28 11:38:17.530021 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:17.529984    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:38:22.535854 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:22.535818    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:38:22.536469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:22.536441    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:38:32.536545 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:32.536504    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:38:42.536913 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:42.536873    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:38:52.536498 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:38:52.536417    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:39:02.536879 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:02.536834    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:39:12.537140 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:12.537102    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:39:22.536860 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:22.536815    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.83:8080: connect: connection refused"
Jul 28 11:39:32.537598 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:32.537568    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:39:33.136783 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:33.136749    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:39:33.137011 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:33.136970    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container" containerID="cri-o://102e754f697d8a5d8471c2fd9b4aec7d1ca43209d02c61509b9a1677d12776b2" gracePeriod=30
Jul 28 11:39:33.137102 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:33.137021    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kube-rbac-proxy" containerID="cri-o://209bf9a419e3e6e4c57647449ba8267542123c431772f271ead3cc04ecb54487" gracePeriod=30
Jul 28 11:39:33.837796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:33.837759    2569 generic.go:356] "Generic (PLEG): container finished" podID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerID="209bf9a419e3e6e4c57647449ba8267542123c431772f271ead3cc04ecb54487" exitCode=2
Jul 28 11:39:33.837796 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:33.837801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerDied","Data":"209bf9a419e3e6e4c57647449ba8267542123c431772f271ead3cc04ecb54487"}
Jul 28 11:39:34.185064 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185027    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:34.185358 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185346    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.185404 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185360    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.185404 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185373    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.185404 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185379    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.185503 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185439    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.185503 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.185449    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="126c85a2-37ba-46ab-8692-27d2378c0973" containerName="storage-initializer"
Jul 28 11:39:34.188485 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.188467    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.191090 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.191067    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 11:39:34.191267 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.191164    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 11:39:34.204048 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.204022    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:34.287042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.287007    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.287042 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.287053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5hq\" (UniqueName: \"kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.287276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.287076    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.287276 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.287195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.388610 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.388571    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.388814 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.388620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5hq\" (UniqueName: \"kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.388814 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.388738    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.388904 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.388867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.389160 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.389141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.389440 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.389423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.391043 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.391023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.398213 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.398183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5hq\" (UniqueName: \"kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.505297 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.505191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:34.636582 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.636549    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:34.639736 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:39:34.639703    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a33372d_0edd_470a_825c_30a45ffdfa46.slice/crio-c2b5266045d080650800e66adbd30420042cb36848b59a4a9759aa9f056cf947 WatchSource:0}: Error finding container c2b5266045d080650800e66adbd30420042cb36848b59a4a9759aa9f056cf947: Status 404 returned error can't find the container with id c2b5266045d080650800e66adbd30420042cb36848b59a4a9759aa9f056cf947
Jul 28 11:39:34.844176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.844086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerStarted","Data":"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"}
Jul 28 11:39:34.844176 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:34.844164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerStarted","Data":"c2b5266045d080650800e66adbd30420042cb36848b59a4a9759aa9f056cf947"}
Jul 28 11:39:37.530703 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:37.530636    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.83:8643/healthz\": dial tcp 10.134.0.83:8643: connect: connection refused"
Jul 28 11:39:37.859585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:37.859549    2569 generic.go:356] "Generic (PLEG): container finished" podID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerID="102e754f697d8a5d8471c2fd9b4aec7d1ca43209d02c61509b9a1677d12776b2" exitCode=0
Jul 28 11:39:37.859585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:37.859584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerDied","Data":"102e754f697d8a5d8471c2fd9b4aec7d1ca43209d02c61509b9a1677d12776b2"}
Jul 28 11:39:37.879648 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:37.879621    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:39:38.023469 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023437    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location\") pod \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") "
Jul 28 11:39:38.023661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023480    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert\") pod \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") "
Jul 28 11:39:38.023661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023507    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2v8s\" (UniqueName: \"kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s\") pod \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") "
Jul 28 11:39:38.023661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") pod \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") "
Jul 28 11:39:38.023661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023564    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config\") pod \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\" (UID: \"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8\") "
Jul 28 11:39:38.023929 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023850    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:39:38.023993 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.023968    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:39:38.024065 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.024046    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:39:38.025723 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.025699    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:39:38.025832 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.025767    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s" (OuterVolumeSpecName: "kube-api-access-z2v8s") pod "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" (UID: "9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8"). InnerVolumeSpecName "kube-api-access-z2v8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:39:38.124452 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.124364    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:38.124452 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.124394    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-cabundle-cert\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:38.124452 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.124404    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2v8s\" (UniqueName: \"kubernetes.io/projected/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-api-access-z2v8s\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:38.124452 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.124414    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:38.124452 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.124423    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:38.865643 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.865560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf" event={"ID":"9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8","Type":"ContainerDied","Data":"2ee81fb6a264fcee149fc93ef381d63000611643bb4509dd3f92523be8257b2c"}
Jul 28 11:39:38.865643 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.865603    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"
Jul 28 11:39:38.865643 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.865631    2569 scope.go:117] "RemoveContainer" containerID="209bf9a419e3e6e4c57647449ba8267542123c431772f271ead3cc04ecb54487"
Jul 28 11:39:38.874200 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.874177    2569 scope.go:117] "RemoveContainer" containerID="102e754f697d8a5d8471c2fd9b4aec7d1ca43209d02c61509b9a1677d12776b2"
Jul 28 11:39:38.881456 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.881437    2569 scope.go:117] "RemoveContainer" containerID="cd5335af5f4af1500e1cbef89545010ccc9ea663214a2bcc9b28d3799a44e1c1"
Jul 28 11:39:38.885857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.885836    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:39:38.889335 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:38.889313    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c98bc684c-s6jxf"]
Jul 28 11:39:40.570921 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:40.570888    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" path="/var/lib/kubelet/pods/9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8/volumes"
Jul 28 11:39:41.881576 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:41.881552    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/0.log"
Jul 28 11:39:41.882075 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:41.881588    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerID="9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14" exitCode=1
Jul 28 11:39:41.882075 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:41.881674    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerDied","Data":"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"}
Jul 28 11:39:42.887916 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:42.887890    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/0.log"
Jul 28 11:39:42.888306 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:42.888009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerStarted","Data":"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"}
Jul 28 11:39:44.214168 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:44.214132    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:44.214582 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:44.214327    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer" containerID="cri-o://8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f" gracePeriod=30
Jul 28 11:39:45.552018 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.551994    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/1.log"
Jul 28 11:39:45.552367 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.552352    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/0.log"
Jul 28 11:39:45.554516 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.554501    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:45.687026 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.686989    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config\") pod \"3a33372d-0edd-470a-825c-30a45ffdfa46\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") "
Jul 28 11:39:45.687248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.687041    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx5hq\" (UniqueName: \"kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq\") pod \"3a33372d-0edd-470a-825c-30a45ffdfa46\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") "
Jul 28 11:39:45.687248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.687061    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location\") pod \"3a33372d-0edd-470a-825c-30a45ffdfa46\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") "
Jul 28 11:39:45.687248 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.687089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls\") pod \"3a33372d-0edd-470a-825c-30a45ffdfa46\" (UID: \"3a33372d-0edd-470a-825c-30a45ffdfa46\") "
Jul 28 11:39:45.687433 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.687334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a33372d-0edd-470a-825c-30a45ffdfa46" (UID: "3a33372d-0edd-470a-825c-30a45ffdfa46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:39:45.687433 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.687362    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a33372d-0edd-470a-825c-30a45ffdfa46" (UID: "3a33372d-0edd-470a-825c-30a45ffdfa46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:39:45.689149 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.689120    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a33372d-0edd-470a-825c-30a45ffdfa46" (UID: "3a33372d-0edd-470a-825c-30a45ffdfa46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:39:45.689252 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.689176    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq" (OuterVolumeSpecName: "kube-api-access-dx5hq") pod "3a33372d-0edd-470a-825c-30a45ffdfa46" (UID: "3a33372d-0edd-470a-825c-30a45ffdfa46"). InnerVolumeSpecName "kube-api-access-dx5hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:39:45.787634 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.787597    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:45.787634 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.787627    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5hq\" (UniqueName: \"kubernetes.io/projected/3a33372d-0edd-470a-825c-30a45ffdfa46-kube-api-access-dx5hq\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:45.787634 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.787640    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a33372d-0edd-470a-825c-30a45ffdfa46-kserve-provision-location\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:45.787901 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.787654    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a33372d-0edd-470a-825c-30a45ffdfa46-proxy-tls\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:39:45.900924 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.900896    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/1.log"
Jul 28 11:39:45.901309 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901291    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6_3a33372d-0edd-470a-825c-30a45ffdfa46/storage-initializer/0.log"
Jul 28 11:39:45.901405 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901329    2569 generic.go:356] "Generic (PLEG): container finished" podID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerID="8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f" exitCode=1
Jul 28 11:39:45.901467 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901405    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"
Jul 28 11:39:45.901523 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901401    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerDied","Data":"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"}
Jul 28 11:39:45.901585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6" event={"ID":"3a33372d-0edd-470a-825c-30a45ffdfa46","Type":"ContainerDied","Data":"c2b5266045d080650800e66adbd30420042cb36848b59a4a9759aa9f056cf947"}
Jul 28 11:39:45.901585 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.901559    2569 scope.go:117] "RemoveContainer" containerID="8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"
Jul 28 11:39:45.910261 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.910243    2569 scope.go:117] "RemoveContainer" containerID="9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"
Jul 28 11:39:45.917606 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.917591    2569 scope.go:117] "RemoveContainer" containerID="8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"
Jul 28 11:39:45.917890 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:39:45.917869    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f\": container with ID starting with 8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f not found: ID does not exist" containerID="8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"
Jul 28 11:39:45.917949 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.917902    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f"} err="failed to get container status \"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f\": rpc error: code = NotFound desc = could not find container \"8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f\": container with ID starting with 8120351d99545025f4303b3b0afe232335ab3d19833aa28de931ffa6c984976f not found: ID does not exist"
Jul 28 11:39:45.917949 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.917927    2569 scope.go:117] "RemoveContainer" containerID="9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"
Jul 28 11:39:45.918150 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:39:45.918133    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14\": container with ID starting with 9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14 not found: ID does not exist" containerID="9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"
Jul 28 11:39:45.918208 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.918160    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14"} err="failed to get container status \"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14\": rpc error: code = NotFound desc = could not find container \"9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14\": container with ID starting with 9f770eb1dd9bb00573c332badb769dec2c5c03dbe20be34ebbbbae124e292d14 not found: ID does not exist"
Jul 28 11:39:45.938222 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.938136    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:45.941611 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:45.941586    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5c565c778f-jfkz6"]
Jul 28 11:39:46.527095 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527059    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zqmgv/must-gather-q9tpv"]
Jul 28 11:39:46.527386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527374    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="storage-initializer"
Jul 28 11:39:46.527431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527388    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="storage-initializer"
Jul 28 11:39:46.527431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527398    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.527431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527403    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.527431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527411    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kube-rbac-proxy"
Jul 28 11:39:46.527431 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527417    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kube-rbac-proxy"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527434    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527439    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527445    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527451    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527499    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kserve-container"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527505    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1cc2b1-3acd-48dd-8da9-64b0a3bb70a8" containerName="kube-rbac-proxy"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527513    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.527583 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.527519    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" containerName="storage-initializer"
Jul 28 11:39:46.531789 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.531767    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.537811 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.537783    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zqmgv\"/\"openshift-service-ca.crt\""
Jul 28 11:39:46.537955 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.537839    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zqmgv\"/\"kube-root-ca.crt\""
Jul 28 11:39:46.566807 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.566777    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a33372d-0edd-470a-825c-30a45ffdfa46" path="/var/lib/kubelet/pods/3a33372d-0edd-470a-825c-30a45ffdfa46/volumes"
Jul 28 11:39:46.567268 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.567029    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zqmgv/must-gather-q9tpv"]
Jul 28 11:39:46.694992 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.694963    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.695180 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.695095    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjg7\" (UniqueName: \"kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.796472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.796378    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjg7\" (UniqueName: \"kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.796472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.796449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.796843 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.796823    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.805013 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.804988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjg7\" (UniqueName: \"kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7\") pod \"must-gather-q9tpv\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") " pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.850975 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.850944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:39:46.980352 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.980241    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zqmgv/must-gather-q9tpv"]
Jul 28 11:39:46.982961 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:39:46.982929    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdeaf737_b51d_4965_bbd1_a66fffaf3089.slice/crio-0c26afc746b38393830a84d2537bdc8060e43fb97a795951204775d8f1a6ae45 WatchSource:0}: Error finding container 0c26afc746b38393830a84d2537bdc8060e43fb97a795951204775d8f1a6ae45: Status 404 returned error can't find the container with id 0c26afc746b38393830a84d2537bdc8060e43fb97a795951204775d8f1a6ae45
Jul 28 11:39:46.984553 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:46.984537    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:39:47.915447 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:47.915350    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" event={"ID":"fdeaf737-b51d-4965-bbd1-a66fffaf3089","Type":"ContainerStarted","Data":"0c26afc746b38393830a84d2537bdc8060e43fb97a795951204775d8f1a6ae45"}
Jul 28 11:39:53.947386 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:53.947326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" event={"ID":"fdeaf737-b51d-4965-bbd1-a66fffaf3089","Type":"ContainerStarted","Data":"7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753"}
Jul 28 11:39:53.947866 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:53.947399    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" event={"ID":"fdeaf737-b51d-4965-bbd1-a66fffaf3089","Type":"ContainerStarted","Data":"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"}
Jul 28 11:39:53.967402 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:39:53.967350    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" podStartSLOduration=1.874625048 podStartE2EDuration="7.967332196s" totalImagesPullingTime="6.092707148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:39:53.964826286 +0000 UTC m=+5551.885530340" watchObservedRunningTime="2026-07-28 11:39:53.967332196 +0000 UTC m=+5551.888036280"
Jul 28 11:40:20.067088 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:20.067052    2569 generic.go:356] "Generic (PLEG): container finished" podID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerID="d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc" exitCode=0
Jul 28 11:40:20.067621 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:20.067130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" event={"ID":"fdeaf737-b51d-4965-bbd1-a66fffaf3089","Type":"ContainerDied","Data":"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"}
Jul 28 11:40:20.067621 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:20.067541    2569 scope.go:117] "RemoveContainer" containerID="d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"
Jul 28 11:40:20.998797 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:20.998766    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zqmgv_must-gather-q9tpv_fdeaf737-b51d-4965-bbd1-a66fffaf3089/gather/0.log"
Jul 28 11:40:21.677544 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.677503    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x6kqb/must-gather-fghpx"]
Jul 28 11:40:21.681103 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.681081    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.683663 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.683643    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x6kqb\"/\"default-dockercfg-26d66\""
Jul 28 11:40:21.683937 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.683919    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6kqb\"/\"openshift-service-ca.crt\""
Jul 28 11:40:21.684724 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.684711    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6kqb\"/\"kube-root-ca.crt\""
Jul 28 11:40:21.688870 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.688846    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6kqb/must-gather-fghpx"]
Jul 28 11:40:21.799130 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.799090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2eed32fb-a565-4af9-b4c0-2494467cb43a-must-gather-output\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.799319 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.799181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sq8g\" (UniqueName: \"kubernetes.io/projected/2eed32fb-a565-4af9-b4c0-2494467cb43a-kube-api-access-2sq8g\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.899765 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.899726    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sq8g\" (UniqueName: \"kubernetes.io/projected/2eed32fb-a565-4af9-b4c0-2494467cb43a-kube-api-access-2sq8g\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.899967 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.899783    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2eed32fb-a565-4af9-b4c0-2494467cb43a-must-gather-output\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.900113 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.900094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2eed32fb-a565-4af9-b4c0-2494467cb43a-must-gather-output\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.909784 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.909754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sq8g\" (UniqueName: \"kubernetes.io/projected/2eed32fb-a565-4af9-b4c0-2494467cb43a-kube-api-access-2sq8g\") pod \"must-gather-fghpx\" (UID: \"2eed32fb-a565-4af9-b4c0-2494467cb43a\") " pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:21.990777 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:21.990672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6kqb/must-gather-fghpx"
Jul 28 11:40:22.122871 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:22.122839    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6kqb/must-gather-fghpx"]
Jul 28 11:40:22.125963 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:40:22.125931    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eed32fb_a565_4af9_b4c0_2494467cb43a.slice/crio-6162113f58c15e4456fe875f1bec326c99ba87075327c89aa7bc9eed1935536d WatchSource:0}: Error finding container 6162113f58c15e4456fe875f1bec326c99ba87075327c89aa7bc9eed1935536d: Status 404 returned error can't find the container with id 6162113f58c15e4456fe875f1bec326c99ba87075327c89aa7bc9eed1935536d
Jul 28 11:40:23.084077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:23.084035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6kqb/must-gather-fghpx" event={"ID":"2eed32fb-a565-4af9-b4c0-2494467cb43a","Type":"ContainerStarted","Data":"948422b1d09b9f2bb48f0073399dee61eff92855a14dca9904c16bb1981d99a0"}
Jul 28 11:40:23.084473 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:23.084087    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6kqb/must-gather-fghpx" event={"ID":"2eed32fb-a565-4af9-b4c0-2494467cb43a","Type":"ContainerStarted","Data":"a40c0509643a0aa924665fa241d125fadaab26ed5eca6c9149e543c9984dfaf8"}
Jul 28 11:40:23.084473 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:23.084104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6kqb/must-gather-fghpx" event={"ID":"2eed32fb-a565-4af9-b4c0-2494467cb43a","Type":"ContainerStarted","Data":"6162113f58c15e4456fe875f1bec326c99ba87075327c89aa7bc9eed1935536d"}
Jul 28 11:40:23.102489 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:23.102436    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x6kqb/must-gather-fghpx" podStartSLOduration=1.3652341319999999 podStartE2EDuration="2.102418825s" totalImagesPullingTime="737.184693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:40:23.100500846 +0000 UTC m=+5581.021204899" watchObservedRunningTime="2026-07-28 11:40:23.102418825 +0000 UTC m=+5581.023122876"
Jul 28 11:40:24.387408 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:24.387367    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6k7q6_92eeb473-d6b5-437e-a0dd-8a128f8b10d0/global-pull-secret-syncer/0.log"
Jul 28 11:40:24.633130 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:24.633087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vf62c_f38ab969-d6e3-4f7b-9e3b-5ed7dab44cef/konnectivity-agent/0.log"
Jul 28 11:40:24.707227 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:24.707170    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-108.ec2.internal_45b79e16bf677a5e3c62a28e1b4ebba5/haproxy/0.log"
Jul 28 11:40:26.523318 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:26.523274    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-zqmgv/must-gather-q9tpv"]
Jul 28 11:40:26.523847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:26.523495    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-zqmgv/must-gather-q9tpv" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="copy" containerID="cri-o://7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753" gracePeriod=2
Jul 28 11:40:26.531265 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:26.531232    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-zqmgv/must-gather-q9tpv"]
Jul 28 11:40:26.531461 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:40:26.531430    2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fdeaf737-b51d-4965-bbd1-a66fffaf3089\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-zqmgv\"/\"must-gather-q9tpv\": pods \"must-gather-q9tpv\" not found" pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:40:26.922916 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:26.916727    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zqmgv_must-gather-q9tpv_fdeaf737-b51d-4965-bbd1-a66fffaf3089/copy/0.log"
Jul 28 11:40:26.922916 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:26.920354    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:40:27.047344 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.047302    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output\") pod \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") "
Jul 28 11:40:27.047542 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.047397    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfjg7\" (UniqueName: \"kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7\") pod \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\" (UID: \"fdeaf737-b51d-4965-bbd1-a66fffaf3089\") "
Jul 28 11:40:27.050763 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.049357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "fdeaf737-b51d-4965-bbd1-a66fffaf3089" (UID: "fdeaf737-b51d-4965-bbd1-a66fffaf3089"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:40:27.053978 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.052104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7" (OuterVolumeSpecName: "kube-api-access-xfjg7") pod "fdeaf737-b51d-4965-bbd1-a66fffaf3089" (UID: "fdeaf737-b51d-4965-bbd1-a66fffaf3089"). InnerVolumeSpecName "kube-api-access-xfjg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:40:27.106936 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.106901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-zqmgv_must-gather-q9tpv_fdeaf737-b51d-4965-bbd1-a66fffaf3089/copy/0.log"
Jul 28 11:40:27.107369 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.107340    2569 generic.go:356] "Generic (PLEG): container finished" podID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerID="7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753" exitCode=143
Jul 28 11:40:27.107480 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.107413    2569 scope.go:117] "RemoveContainer" containerID="7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753"
Jul 28 11:40:27.107605 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.107586    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zqmgv/must-gather-q9tpv"
Jul 28 11:40:27.125834 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.125807    2569 scope.go:117] "RemoveContainer" containerID="d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"
Jul 28 11:40:27.149036 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.148997    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fdeaf737-b51d-4965-bbd1-a66fffaf3089-must-gather-output\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:40:27.149036 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.149038    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfjg7\" (UniqueName: \"kubernetes.io/projected/fdeaf737-b51d-4965-bbd1-a66fffaf3089-kube-api-access-xfjg7\") on node \"ip-10-0-133-108.ec2.internal\" DevicePath \"\""
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.149626    2569 scope.go:117] "RemoveContainer" containerID="7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753"
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:40:27.150228    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753\": container with ID starting with 7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753 not found: ID does not exist" containerID="7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753"
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.150268    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753"} err="failed to get container status \"7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753\": rpc error: code = NotFound desc = could not find container \"7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753\": container with ID starting with 7631468e64cb5d7f58d7362b784bd516e9d08c4e42d24777a26e942bc6dca753 not found: ID does not exist"
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.150293    2569 scope.go:117] "RemoveContainer" containerID="d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: E0728 11:40:27.150562    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc\": container with ID starting with d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc not found: ID does not exist" containerID="d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"
Jul 28 11:40:27.151857 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:27.150597    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc"} err="failed to get container status \"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc\": rpc error: code = NotFound desc = could not find container \"d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc\": container with ID starting with d83a8bbb9ff0f09aa5f8d49fe9accafe1864a40966fadf62a2b172d76d0960dc not found: ID does not exist"
Jul 28 11:40:28.570100 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:28.569570    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" path="/var/lib/kubelet/pods/fdeaf737-b51d-4965-bbd1-a66fffaf3089/volumes"
Jul 28 11:40:29.562824 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:29.562783    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-847755dcc8-kl6ll_b2b4cb46-8f76-4f9b-b140-3d7af40dd596/metrics-server/0.log"
Jul 28 11:40:29.595519 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:29.595434    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-z5rmx_8683564a-8905-45ba-8bd3-ca7b5531e249/monitoring-plugin/0.log"
Jul 28 11:40:29.816493 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:29.816465    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5hlp_a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9/node-exporter/0.log"
Jul 28 11:40:29.844630 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:29.844590    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5hlp_a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9/kube-rbac-proxy/0.log"
Jul 28 11:40:29.874410 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:29.874258    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w5hlp_a5500cd0-9a88-489b-a1a0-ea66cdc2b9d9/init-textfile/0.log"
Jul 28 11:40:30.003037 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.002982    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/prometheus/0.log"
Jul 28 11:40:30.032641 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.032605    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/config-reloader/0.log"
Jul 28 11:40:30.061073 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.061046    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/thanos-sidecar/0.log"
Jul 28 11:40:30.083995 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.083910    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/kube-rbac-proxy-web/0.log"
Jul 28 11:40:30.108192 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.108164    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/kube-rbac-proxy/0.log"
Jul 28 11:40:30.136281 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.136207    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/kube-rbac-proxy-thanos/0.log"
Jul 28 11:40:30.165085 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.165055    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a917ac49-b6a3-44f4-80fd-4276875ac989/init-config-reloader/0.log"
Jul 28 11:40:30.198702 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.198648    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lxmx4_0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e/prometheus-operator/0.log"
Jul 28 11:40:30.221970 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.221941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lxmx4_0c1d5989-1e2d-4c6a-aaec-be4e767d8e4e/kube-rbac-proxy/0.log"
Jul 28 11:40:30.258007 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.257980    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-mlgl5_94652a70-76f8-40cb-b4b6-cc7030a21069/prometheus-operator-admission-webhook/0.log"
Jul 28 11:40:30.431603 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.431567    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/thanos-query/0.log"
Jul 28 11:40:30.467106 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.467073    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/kube-rbac-proxy-web/0.log"
Jul 28 11:40:30.497243 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.497214    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/kube-rbac-proxy/0.log"
Jul 28 11:40:30.525586 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.525537    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/prom-label-proxy/0.log"
Jul 28 11:40:30.558659 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.558549    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/kube-rbac-proxy-rules/0.log"
Jul 28 11:40:30.589927 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:30.589902    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8c959989b-b5krg_003e27be-6245-47da-87d4-49ed86acf3e7/kube-rbac-proxy-metrics/0.log"
Jul 28 11:40:32.724145 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.724109    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-5vpqg_5fef08c7-1084-4238-b972-baadace7894b/download-server/0.log"
Jul 28 11:40:32.985739 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.985637    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"]
Jul 28 11:40:32.985984 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.985969    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="copy"
Jul 28 11:40:32.985984 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.985985    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="copy"
Jul 28 11:40:32.986077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.985995    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="gather"
Jul 28 11:40:32.986077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.986000    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="gather"
Jul 28 11:40:32.986077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.986051    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="gather"
Jul 28 11:40:32.986077 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:32.986060    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdeaf737-b51d-4965-bbd1-a66fffaf3089" containerName="copy"
Jul 28 11:40:33.001994 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.001963    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"]
Jul 28 11:40:33.002247 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.002075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.114300 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.114264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvxs\" (UniqueName: \"kubernetes.io/projected/53061ba9-77b6-4e8f-8563-17828209b0e5-kube-api-access-hwvxs\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.114472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.114384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-lib-modules\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.114472 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.114441    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-podres\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.114589 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.114474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-proc\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.114589 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.114530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-sys\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215284 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-podres\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-proc\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-sys\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215482 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvxs\" (UniqueName: \"kubernetes.io/projected/53061ba9-77b6-4e8f-8563-17828209b0e5-kube-api-access-hwvxs\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215674 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-lib-modules\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215761 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215737    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-lib-modules\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215897 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-podres\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.215978 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.215911    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-proc\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.216036 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.216001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53061ba9-77b6-4e8f-8563-17828209b0e5-sys\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.224596 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.224570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvxs\" (UniqueName: \"kubernetes.io/projected/53061ba9-77b6-4e8f-8563-17828209b0e5-kube-api-access-hwvxs\") pod \"perf-node-gather-daemonset-k9l5t\" (UID: \"53061ba9-77b6-4e8f-8563-17828209b0e5\") " pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.316232 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.316135    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:33.483385 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:33.483325    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"]
Jul 28 11:40:33.486917 ip-10-0-133-108 kubenswrapper[2569]: W0728 11:40:33.486290    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod53061ba9_77b6_4e8f_8563_17828209b0e5.slice/crio-6f648212af2a08bae74af2d7a45e23bf28666897845b3e9523d6051cd3959307 WatchSource:0}: Error finding container 6f648212af2a08bae74af2d7a45e23bf28666897845b3e9523d6051cd3959307: Status 404 returned error can't find the container with id 6f648212af2a08bae74af2d7a45e23bf28666897845b3e9523d6051cd3959307
Jul 28 11:40:34.150377 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.150333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t" event={"ID":"53061ba9-77b6-4e8f-8563-17828209b0e5","Type":"ContainerStarted","Data":"555f47c1bac03b63a8aded94e787ab3e017d04d75b74bd7b7b2f8d8b4d3476c8"}
Jul 28 11:40:34.150893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.150398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t" event={"ID":"53061ba9-77b6-4e8f-8563-17828209b0e5","Type":"ContainerStarted","Data":"6f648212af2a08bae74af2d7a45e23bf28666897845b3e9523d6051cd3959307"}
Jul 28 11:40:34.150893 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.150432    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:34.370307 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.370277    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv85l_82fe5b6a-818c-4b00-9af5-0133aaebc62b/dns/0.log"
Jul 28 11:40:34.393374 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.393334    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xv85l_82fe5b6a-818c-4b00-9af5-0133aaebc62b/kube-rbac-proxy/0.log"
Jul 28 11:40:34.447267 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:34.447186    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg6wf_395c1740-3932-4e87-bf52-06ee0acdd2b3/dns-node-resolver/0.log"
Jul 28 11:40:35.050953 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:35.050911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wdjkk_04758362-69bc-4f6f-a3e5-8ff5d2373d57/node-ca/0.log"
Jul 28 11:40:36.157840 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:36.157807    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c6x8n_5d005d46-961d-4f0b-8fb1-a678331aa43f/serve-healthcheck-canary/0.log"
Jul 28 11:40:36.664109 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:36.664079    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5z5_5eb49b5a-7beb-4d29-9432-ff5a3c35de8b/kube-rbac-proxy/0.log"
Jul 28 11:40:36.684305 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:36.684261    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5z5_5eb49b5a-7beb-4d29-9432-ff5a3c35de8b/exporter/0.log"
Jul 28 11:40:36.704413 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:36.704382    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fr5z5_5eb49b5a-7beb-4d29-9432-ff5a3c35de8b/extractor/0.log"
Jul 28 11:40:39.404191 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.404159    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2hw6q_e4d8076f-30dd-4054-8c4f-5d35e38cc64c/s3-init/0.log"
Jul 28 11:40:39.476214 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.476179    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2hw6q_e4d8076f-30dd-4054-8c4f-5d35e38cc64c/download-hf-model/0.log"
Jul 28 11:40:39.515898 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.515869    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-w828m_e14a887b-e068-4384-91f6-42bd8c16ca18/s3-tls-init-custom/0.log"
Jul 28 11:40:39.534049 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.534022    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-w828m_e14a887b-e068-4384-91f6-42bd8c16ca18/download-hf-model/0.log"
Jul 28 11:40:39.565653 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.565626    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fqkpm_54cdfa2e-c983-4ac9-a04e-f63bda76d19e/s3-tls-init-serving/0.log"
Jul 28 11:40:39.582985 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.582956    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-fqkpm_54cdfa2e-c983-4ac9-a04e-f63bda76d19e/download-hf-model/0.log"
Jul 28 11:40:39.612133 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.612104    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-gwkp4_358b6648-fa79-49f8-bcb1-f5708ba5aa10/seaweedfs/0.log"
Jul 28 11:40:39.635614 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.635584    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-wt9bz_de01ce05-d9ec-48ad-9ad7-de1038191542/seaweedfs-tls-custom/0.log"
Jul 28 11:40:39.659900 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:39.659828    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-4wplm_04cafd22-b69e-4346-8d76-8e4339203d04/seaweedfs-tls-serving/0.log"
Jul 28 11:40:40.165661 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:40.165631    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t"
Jul 28 11:40:40.183316 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:40.183266    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x6kqb/perf-node-gather-daemonset-k9l5t" podStartSLOduration=8.183250808 podStartE2EDuration="8.183250808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:40:34.16941282 +0000 UTC m=+5592.090116875" watchObservedRunningTime="2026-07-28 11:40:40.183250808 +0000 UTC m=+5598.103954860"
Jul 28 11:40:45.129071 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.129009    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/kube-multus-additional-cni-plugins/0.log"
Jul 28 11:40:45.152863 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.152820    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/egress-router-binary-copy/0.log"
Jul 28 11:40:45.172956 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.172931    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/cni-plugins/0.log"
Jul 28 11:40:45.195124 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.195094    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/bond-cni-plugin/0.log"
Jul 28 11:40:45.215600 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.215571    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/routeoverride-cni/0.log"
Jul 28 11:40:45.236607 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.236575    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/whereabouts-cni-bincopy/0.log"
Jul 28 11:40:45.260847 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.260821    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgh6v_1eaa1fdd-6ff4-4f47-9d3b-d6d00b17c612/whereabouts-cni/0.log"
Jul 28 11:40:45.489158 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.489118    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jt8td_35d00954-6525-4b04-937c-b49920e44a34/kube-multus/0.log"
Jul 28 11:40:45.581426 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.581394    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2r6mj_46632d46-0a7f-4b86-99bd-65ad313076e8/network-metrics-daemon/0.log"
Jul 28 11:40:45.596251 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:45.596223    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2r6mj_46632d46-0a7f-4b86-99bd-65ad313076e8/kube-rbac-proxy/0.log"
Jul 28 11:40:46.653556 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.653518    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-controller/0.log"
Jul 28 11:40:46.670734 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.670708    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/0.log"
Jul 28 11:40:46.670734 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.670720    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovn-acl-logging/1.log"
Jul 28 11:40:46.689387 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.689360    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/kube-rbac-proxy-node/0.log"
Jul 28 11:40:46.799495 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.799468    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 11:40:46.819009 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.818979    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/northd/0.log"
Jul 28 11:40:46.839264 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.839233    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/nbdb/0.log"
Jul 28 11:40:46.861463 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.861439    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/sbdb/0.log"
Jul 28 11:40:46.947981 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:46.947901    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kvnkg_9a2d1abe-36a3-4e30-a01e-a238eba3decb/ovnkube-controller/0.log"
Jul 28 11:40:48.193489 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:48.193457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4rm74_56b512b3-61eb-4eb5-9a48-1ea758c47797/network-check-target-container/0.log"
Jul 28 11:40:49.201848 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:49.201817    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-42b4h_cddee4ee-e652-477b-838b-b04d06767bff/iptables-alerter/0.log"
Jul 28 11:40:49.863071 ip-10-0-133-108 kubenswrapper[2569]: I0728 11:40:49.863036    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-l5m7h_e954e4b3-8d05-41ee-b3fc-12e312fb6e97/tuned/0.log"