Jul 29 10:29:07.963514 ip-10-0-134-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:29:08.419196 ip-10-0-134-91 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:29:08.419196 ip-10-0-134-91 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:29:08.419196 ip-10-0-134-91 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:29:08.419196 ip-10-0-134-91 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:29:08.419196 ip-10-0-134-91 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:29:08.420405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.420292    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:29:08.426465 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426437    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.426465 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426460    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.426465 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426466    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.426465 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426470    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.426465 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426475    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426480    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426485    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426491    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426495    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426500    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426504    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426509    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426513    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426519    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426523    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426528    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426532    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426537    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426541    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426546    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426550    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426557    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426562    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.426811 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426566    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426572    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426577    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426582    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426586    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426590    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426595    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426599    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426604    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426609    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426613    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426617    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426622    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426626    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426632    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426637    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426642    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426648    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426653    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.427700 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426658    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426666    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426673    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426679    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426684    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426692    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426698    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426703    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426711    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426716    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426722    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426726    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426733    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426739    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426747    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426753    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426761    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426768    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.428611 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426775    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426780    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426786    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426790    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426798    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426804    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426808    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426815    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426821    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426827    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426832    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426838    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426842    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426848    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426853    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426858    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426863    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426868    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426873    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426878    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.429618 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426884    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426889    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426894    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426899    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426904    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426909    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426914    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426921    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426927    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426931    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426935    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426940    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426945    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426949    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426954    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426959    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426964    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426968    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.426973    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428355    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428369    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.430621 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428375    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428381    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428386    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428391    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428396    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428402    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428407    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428412    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428417    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428422    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428427    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428431    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428436    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428441    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428446    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428451    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428455    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428460    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428466    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428472    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.431600 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428477    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428482    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428487    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428491    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428497    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428502    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428507    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428511    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428517    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428521    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428526    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428534    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428541    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428547    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428552    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428557    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428562    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428566    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428573    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.432444 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428577    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428583    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428587    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428592    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428597    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428601    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428606    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428610    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428615    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428620    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428624    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428631    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428638    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428644    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428650    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428656    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428664    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428670    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428676    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.432970 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428682    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428687    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428693    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428698    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428703    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428710    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428715    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428720    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428725    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428731    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428736    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428740    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428745    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428749    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428754    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428759    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428763    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428768    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428773    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428777    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.433593 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428782    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428786    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428791    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428795    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428800    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428805    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428810    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428815    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428820    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428826    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428831    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428835    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428840    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428846    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428851    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428856    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428862    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428867    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.428871    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429702    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.434413 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429713    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429719    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429724    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429729    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429738    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429744    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429752    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429758    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429764    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429769    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429774    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429779    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429784    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429789    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429794    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429799    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429804    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429809    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429816    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.434953 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429824    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429830    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429836    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429841    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429846    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429852    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429858    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429864    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429869    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429875    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429880    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429885    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429889    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429894    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429899    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429904    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429909    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429915    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429920    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429924    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.435513 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429929    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429934    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429938    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429943    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429948    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429954    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429959    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429964    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429969    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429974    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429978    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429983    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429987    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429993    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.429998    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430003    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430007    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430012    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430017    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430022    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.436134 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430027    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430032    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430036    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430041    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430045    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430050    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430055    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430061    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430066    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430071    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430076    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430081    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430086    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430090    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430096    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430100    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430106    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430111    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430116    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430121    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.436736 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430126    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430131    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430136    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430143    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430149    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430154    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430160    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430165    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430169    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430174    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430180    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430188    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430195    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430221    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430227    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430232    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430237    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430241    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.430246    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430371    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:29:08.437340 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430392    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430406    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430415    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430424    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430431    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430439    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430447    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430454    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430467    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430474    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430481    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430487    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430493    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430498    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430504    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430510    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430516    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430523    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430529    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430536    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430543    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430549    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430557    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:29:08.437873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430563    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430568    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430575    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430580    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430586    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430592    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430598    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430604    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430612    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430617    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430623    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430628    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430633    2551 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430638    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430648    2551 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430656    2551 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430662    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430669    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430678    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430685    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430691    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430697    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430703    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430709    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430715    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:29:08.438498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430721    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430727    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430733    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430739    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430746    2551 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430754    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430760    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430767    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430773    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430779    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430784    2551 flags.go:64] FLAG: --help="false"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430790    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430796    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430802    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430808    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430814    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430821    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430826    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430831    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430837    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430843    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430849    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430855    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430860    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:29:08.439281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430868    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430875    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430882    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430887    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430893    2551 flags.go:64] FLAG: --lock-file=""
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430899    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430904    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430910    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430920    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430926    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430932    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430938    2551 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430943    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430951    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430957    2551 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430963    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430972    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430978    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430986    2551 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430992    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.430998    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431003    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431009    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431015    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431021    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:29:08.440170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431027    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431040    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431046    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431052    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431057    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431063    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431073    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431081    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431087    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431096    2551 flags.go:64] FLAG: --port="10250"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431103    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431109    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e50e5c2f6cdb05f"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431115    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431121    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431127    2551 flags.go:64] FLAG: --register-node="true"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431133    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431140    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431146    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431152    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431157    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431165    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431173    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431179    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431185    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431190    2551 flags.go:64] FLAG: --runonce="false"
Jul 29 10:29:08.441069 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431196    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431221    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431228    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431233    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431239    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431245    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431251    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431257    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431263    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431268    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431274    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431279    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431285    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431291    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431298    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431308    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431316    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431322    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431330    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431335    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431341    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431346    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431352    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431358    2551 flags.go:64] FLAG: --v="2"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431367    2551 flags.go:64] FLAG: --version="false"
Jul 29 10:29:08.441768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431375    2551 flags.go:64] FLAG: --vmodule=""
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431382    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.431388    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431627    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431638    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431644    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431650    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431656    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431661    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431667    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431671    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431677    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431682    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431687    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431692    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431697    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431701    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431706    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431711    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431716    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.442459 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431721    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431729    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431733    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431741    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431747    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431752    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431757    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431761    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431766    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431771    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431776    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431783    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431790    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431796    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431802    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431807    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431813    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431820    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431826    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.442987 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431831    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431836    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431841    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431846    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431851    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431856    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431862    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431867    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431872    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431879    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431886    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431891    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431896    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431901    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431908    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431914    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431922    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431927    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431932    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.443567 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431937    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431942    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431947    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431952    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431957    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431962    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431967    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431972    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431978    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431982    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431987    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431992    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.431998    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432004    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432009    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432014    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432020    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432025    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432030    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432035    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.444087 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432040    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432045    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432050    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432055    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432061    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432066    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432071    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432078    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432083    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432094    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432100    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432105    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432111    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432117    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432122    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432128    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432133    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432137    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432142    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.444643 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432147    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432152    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432157    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432162    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.432167    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.432920    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.442279    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.442302    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.442325    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.442332    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442416    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442422    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442426    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442429    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442433    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.445148 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442436    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442440    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442443    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442447    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442450    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442453    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442456    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442460    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442463    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442466    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442469    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442472    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442476    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442479    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442482    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442485    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442488    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442490    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442494    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442497    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.445608 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442500    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442502    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442511    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442514    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442517    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442520    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442523    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442526    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442529    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442532    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442535    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442538    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442540    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442544    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442546    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442549    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442552    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442555    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442558    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.446143 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442562    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442567    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442571    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442573    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442576    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442579    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442582    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442585    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442589    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442593    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442596    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442599    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442602    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442605    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442608    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442611    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442614    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442617    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442620    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442623    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.446681 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442627    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442630    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442633    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442636    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442638    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442642    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442645    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442648    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442650    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442653    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442656    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442659    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442661    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442664    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442667    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442670    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442673    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442676    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442679    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442682    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.447276 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442685    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442687    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442692    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442695    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442698    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442702    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442705    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442708    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442711    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442716    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442719    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442722    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442726    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442729    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442732    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.447834 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.442737    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442872    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442878    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442881    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442884    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442887    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442890    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442893    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442896    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442899    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442901    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442904    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442907    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442911    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442913    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442916    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442919    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442923    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442926    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442929    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:29:08.448275 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442932    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442934    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442937    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442940    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442943    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442946    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442949    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442952    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442955    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442958    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442961    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442965    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442969    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442973    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442976    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442979    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442982    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442984    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442987    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:29:08.448942 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442990    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442994    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.442997    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443000    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443003    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443005    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443008    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443013    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443015    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443019    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443022    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443025    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443028    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443031    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443035    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443038    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443041    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443044    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443047    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443050    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:29:08.449558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443053    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443055    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443058    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443061    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443065    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443069    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443073    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443076    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443079    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443082    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443085    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443088    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443091    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443094    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443097    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443099    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443102    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443105    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443108    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:29:08.450223 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443110    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443114    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443117    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443120    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443122    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443125    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443128    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443131    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443134    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443137    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443140    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443142    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443145    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443148    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443151    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443153    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443156    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443159    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443161    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443164    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:29:08.450733 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443167    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:08.443170    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.443176    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.443341    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.446996    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.448482    2551 server.go:1019] "Starting client certificate rotation"
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.448610    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.448670    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:29:08.451435 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.451334    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:29:08.452733 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.452710    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:29:08.464568 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.464534    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:29:08.467874 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.467853    2551 log.go:25] "Validated CRI v1 image API"
Jul 29 10:29:08.468855 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.468835    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:29:08.474360 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.474326    2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c32f9f7e-eb77-4dda-b27a-02993e5dba08:/dev/nvme0n1p4 d5efaa0f-d6fa-49a4-ac06-1fd144cbb72d:/dev/nvme0n1p3]
Jul 29 10:29:08.474458 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.474358    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:29:08.482169 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.482016    2551 manager.go:216] Machine: {Timestamp:2026-07-29 10:29:08.4802617 +0000 UTC m=+0.382490091 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121703 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ffbe58ac277a87a069abb48f076d SystemUUID:ec27ffbe-58ac-277a-87a0-69abb48f076d BootID:981ee23b-eea8-4b13-a213-dd6724a70541 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7f:5c:41:fb:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7f:5c:41:fb:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:2e:14:9f:f2:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:29:08.482169 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.482142    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:29:08.482407 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.482308    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:29:08.485895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.485856    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:29:08.486070 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.485900    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:29:08.487270 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.487255    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:29:08.487317 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.487274    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:29:08.487317 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.487283    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:29:08.488047 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.488032    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:29:08.489577 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.489562    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:29:08.492934 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.492902    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:29:08.493080 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493069    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:29:08.493127 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493103    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:29:08.493127 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493114    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:29:08.493231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493130    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:29:08.493231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493141    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:29:08.493231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493150    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:29:08.493572 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493559    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:29:08.493615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.493588    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:29:08.494771 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.494745    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:29:08.495914 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.495895    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:29:08.497331 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.497315    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:29:08.497407 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.497348    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:29:08.498321 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498309    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498327    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498333    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498339    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498348    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498354    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:29:08.498362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498362    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:29:08.498526 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498369    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:29:08.498526 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498377    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:29:08.498526 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498384    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:29:08.498526 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498393    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:29:08.498526 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.498399    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:29:08.499285 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.499275    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:29:08.499285 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.499285    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:29:08.503052 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.503032    2551 server.go:1268] "Started kubelet"
Jul 29 10:29:08.503231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.503152    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:29:08.503306 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.503263    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:29:08.503438 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.503375    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:29:08.504274 ip-10-0-134-91 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:29:08.505192 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.505098    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:29:08.505192 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.505118    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:29:08.505192 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.505173    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:29:08.505398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.505194    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:29:08.505398 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.505209    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:29:08.509458 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.508397    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-91.ec2.internal.18c6bbf19195b270  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-91.ec2.internal,UID:ip-10-0-134-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-91.ec2.internal,},FirstTimestamp:2026-07-29 10:29:08.502999664 +0000 UTC m=+0.405228055,LastTimestamp:2026-07-29 10:29:08.502999664 +0000 UTC m=+0.405228055,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-91.ec2.internal,}"
Jul 29 10:29:08.510542 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.510522    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:29:08.513675 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.513660    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:29:08.513758 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.513678    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:29:08.513834 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.513814    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.514392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.514371    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:29:08.514559 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.514543    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:29:08.514675 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.514563    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:29:08.514927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.514904    2551 factory.go:55] Registering systemd factory
Jul 29 10:29:08.514987 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.514934    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:29:08.515234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515218    2551 factory.go:153] Registering CRI-O factory
Jul 29 10:29:08.515297 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515238    2551 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:29:08.515655 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515618    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:29:08.515713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515657    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:29:08.515713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515668    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:29:08.515713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515671    2551 factory.go:103] Registering Raw factory
Jul 29 10:29:08.515713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.515692    2551 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:29:08.516485 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.516465    2551 manager.go:349] Starting recovery of all containers
Jul 29 10:29:08.527534 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.527499    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sztmz"
Jul 29 10:29:08.528911 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.528708    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:29:08.529145 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.528717    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:29:08.529145 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.528807    2551 manager.go:354] Recovery completed
Jul 29 10:29:08.530912 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.530877    2551 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 29 10:29:08.534395 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.534323    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.536535 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.536508    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.536632 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.536544    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.536632 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.536555    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.536732 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.536676    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sztmz"
Jul 29 10:29:08.537397 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.537368    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:29:08.537397 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.537387    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:29:08.537397 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.537387    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.537595 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.537405    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:29:08.538246 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.538224    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.538334 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.538261    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.538334 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.538279    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.539082 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.539070    2551 policy_none.go:49] "None policy: Start"
Jul 29 10:29:08.539120 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.539088    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:29:08.539120 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.539100    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:29:08.540294 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.540279    2551 policy_none.go:47] "Start"
Jul 29 10:29:08.574949 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.574923    2551 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:29:08.575115 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.574977    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:29:08.575115 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.574990    2551 server.go:87] "Starting device plugin registration server"
Jul 29 10:29:08.575814 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.575799    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:29:08.575858 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.575815    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:29:08.575968 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.575945    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:29:08.576086 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.576047    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:29:08.576086 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.576057    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:29:08.577149 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.577111    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:29:08.577250 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.577194    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.676086 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.675990    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.677616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.677256    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.677616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.677289    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.677616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.677300    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.677616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.677329    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.686019 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.685993    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.686019 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.686025    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-91.ec2.internal\": node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.702800 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.702768    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.720944 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.720903    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:29:08.722333 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.722304    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:29:08.722451 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.722347    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:29:08.722451 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.722376    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:29:08.722451 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.722425    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:29:08.724650 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.724628    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:29:08.807267 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.807217    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.823413 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.823382    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"]
Jul 29 10:29:08.823528 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.823471    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.831944 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.831920    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.832038 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.831962    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.832038 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.831978    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.833893 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.833876    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.834075 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834059    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.834128 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834119    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.834875 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834853    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.834982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834891    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.834982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834903    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.834982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834853    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.834982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834966    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.835133 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.834982    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.836388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.836372    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.836460 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.836409    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:29:08.837618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.837591    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:29:08.837618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.837623    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:29:08.837778 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.837636    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:29:08.858060 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.858024    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-91.ec2.internal\" not found" node="ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.863393 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.863371    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-91.ec2.internal\" not found" node="ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.907938 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:08.907887    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:08.918506 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.918467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.918650 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.918510    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e25e3ac4c36569f4c5a0d44ba195e649-config\") pod \"kube-apiserver-proxy-ip-10-0-134-91.ec2.internal\" (UID: \"e25e3ac4c36569f4c5a0d44ba195e649\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:08.918650 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:08.918538    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.008272 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.008124    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:09.019417 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.019536 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e25e3ac4c36569f4c5a0d44ba195e649-config\") pod \"kube-apiserver-proxy-ip-10-0-134-91.ec2.internal\" (UID: \"e25e3ac4c36569f4c5a0d44ba195e649\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.019536 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019446    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.019536 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019514    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e25e3ac4c36569f4c5a0d44ba195e649-config\") pod \"kube-apiserver-proxy-ip-10-0-134-91.ec2.internal\" (UID: \"e25e3ac4c36569f4c5a0d44ba195e649\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.019655 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019600    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.019655 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.019642    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45946e5e38ff9e43b00ea351a5826bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal\" (UID: \"45946e5e38ff9e43b00ea351a5826bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.113271 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.113224    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:09.163409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.163366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.166872 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.166831    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.214088 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.214035    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:09.321612 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.321504    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:09.421913 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.421872    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-91.ec2.internal\" not found"
Jul 29 10:29:09.449124 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.449081    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:29:09.449309 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.449288    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:29:09.467678 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.467649    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:29:09.494768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.494718    2551 apiserver.go:52] "Watching apiserver"
Jul 29 10:29:09.504578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.504546    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:29:09.505679 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.505650    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf","openshift-cluster-node-tuning-operator/tuned-vh8lq","openshift-dns/node-resolver-rwsrm","openshift-multus/multus-additional-cni-plugins-xq7m7","openshift-multus/multus-h45jj","openshift-multus/network-metrics-daemon-9tsjs","openshift-network-operator/iptables-alerter-f7fj6","kube-system/global-pull-secret-syncer-p5v7t","kube-system/konnectivity-agent-979pr","openshift-image-registry/node-ca-f8mdt","openshift-network-diagnostics/network-check-target-89sxt","openshift-ovn-kubernetes/ovnkube-node-k49lk"]
Jul 29 10:29:09.507343 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.507322    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.508567 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.508537    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.510031 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.510010    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.510303 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.510281    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:29:09.510432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.510354    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-djmrp\""
Jul 29 10:29:09.510432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.510364    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.510634 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.510617    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.511031 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.511011    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.511103 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.511086    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5b9vd\""
Jul 29 10:29:09.511166 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.511109    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.511516 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.511501    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.512323 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.512303    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.512449 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.512433    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vjll8\""
Jul 29 10:29:09.512504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.512491    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.512864 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.512848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.513854 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.513826    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.513969 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.513836    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:29:09.513969 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.513943    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:29:09.514084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514060    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:29:09.514256 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514237    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.514343 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514325    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.514406 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.514342    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:09.514406 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514385    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:29:09.514505 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514441    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.514552 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.514389    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvhpx\""
Jul 29 10:29:09.515306 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.515289    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:29:09.515386 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.515312    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8mxb6\""
Jul 29 10:29:09.515775 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.515752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.517099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.517080    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.517216 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.517136    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:09.518224 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.518194    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.518816 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.518795    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.518816 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.518810    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:29:09.518982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.518892    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.519036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.519023    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kdkqb\""
Jul 29 10:29:09.521190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.521160    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.521792 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.521772    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mp7vz\""
Jul 29 10:29:09.521996 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.521972    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-lib-modules\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.521996 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.521990    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:29:09.522162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522014    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-os-release\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.522162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522039    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:29:09.522162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522046    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.522162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522104    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-conf-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.522162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522153    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-multus-certs\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522211    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522280    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-run\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522322    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-etc-tuned\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/76693f47-85b7-4686-aac5-1c24d6ba4627-kube-api-access-zfktq\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.522388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-modprobe-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-conf\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-device-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-k8s-cni-cncf-io\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kzw\" (UniqueName: \"kubernetes.io/projected/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-kube-api-access-55kzw\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-cnibin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522553    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76693f47-85b7-4686-aac5-1c24d6ba4627-tmp-dir\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522583    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.522622 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522609    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76693f47-85b7-4686-aac5-1c24d6ba4627-hosts-file\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd683971-80ad-4673-9fd9-1e1fdd69cda4-host-slash\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522705    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-cnibin\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522721    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522752    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswgh\" (UniqueName: \"kubernetes.io/projected/742e70a2-c439-4669-b404-fb48d7c4a419-kube-api-access-bswgh\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522816    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-netns\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522847    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktwt\" (UniqueName: \"kubernetes.io/projected/375473e0-3d80-476e-8c4e-84205706af05-kube-api-access-tktwt\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-bin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-kubelet\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522926    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj29l\" (UniqueName: \"kubernetes.io/projected/9154bc9f-fb46-4c80-a45c-a886603cddee-kube-api-access-wj29l\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523001 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.522955    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-host\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523014    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd683971-80ad-4673-9fd9-1e1fdd69cda4-iptables-alerter-script\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523053    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-system-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523115    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-etc-kubernetes\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523152    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-var-lib-kubelet\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523172    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-system-cni-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-os-release\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523253    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-hostroot\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523272    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/b78a69bb-c901-48fb-afc4-335eacdd79ec-kube-api-access-t4cz8\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523287    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-socket-dir-parent\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523344    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523367    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-systemd\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523405    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-cni-binary-copy\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.523517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523435    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-multus\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.523432    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523475    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-tmp\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523497    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnfbq\" (UniqueName: \"kubernetes.io/projected/dd683971-80ad-4673-9fd9-1e1fdd69cda4-kube-api-access-vnfbq\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523512    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-daemon-config\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523527    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysconfig\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523549    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-kubernetes\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-sys\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.523940    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.524023    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xvffm\""
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.524053    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.524098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.524053    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.524714 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.524694    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.526412 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.526392    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:29:09.526518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.526475    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"
Jul 29 10:29:09.527126 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527106    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal"]
Jul 29 10:29:09.527880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527477    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:29:09.527880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527525    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:29:09.527880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527594    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kmp8j\""
Jul 29 10:29:09.527880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527754    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:29:09.527880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527815    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:29:09.528193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.527984    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:29:09.528193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.528056    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:29:09.532685 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.532663    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:29:09.533833 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.533813    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal"]
Jul 29 10:29:09.533908 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.533837    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:29:09.541318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.541258    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:24:09 +0000 UTC" deadline="2028-03-09 13:47:21.752793894 +0000 UTC"
Jul 29 10:29:09.541318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.541317    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14139h18m12.211481828s"
Jul 29 10:29:09.553680 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.553656    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsscl"
Jul 29 10:29:09.562799 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.562770    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsscl"
Jul 29 10:29:09.615324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.615295    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:29:09.624301 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624271    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-os-release\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.624402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.624402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624329    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-conf-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.624402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624353    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-multus-certs\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.624402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624385    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-ovn\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.624402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624393    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-os-release\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624414    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624417    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-conf-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624454    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-var-lib-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624493    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-node-log\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.624535    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624552    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-multus-certs\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-run\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.624650    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:10.124599648 +0000 UTC m=+2.026828045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-etc-tuned\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624683    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-run\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/76693f47-85b7-4686-aac5-1c24d6ba4627-kube-api-access-zfktq\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.624777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624820    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-modprobe-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624849    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-conf\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624877    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-log-socket\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624898    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-device-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624921    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-k8s-cni-cncf-io\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624948    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kzw\" (UniqueName: \"kubernetes.io/projected/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-kube-api-access-55kzw\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.624995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625054    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-cnibin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625074    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-k8s-cni-cncf-io\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625081    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-systemd-units\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625128    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625135    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-conf\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-config\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625219    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76693f47-85b7-4686-aac5-1c24d6ba4627-tmp-dir\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625288    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625326    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.625621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625325    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysctl-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-device-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625340    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-cnibin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625374    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625391    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-modprobe-d\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625420    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76693f47-85b7-4686-aac5-1c24d6ba4627-hosts-file\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-sys-fs\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd683971-80ad-4673-9fd9-1e1fdd69cda4-host-slash\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625511    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/76693f47-85b7-4686-aac5-1c24d6ba4627-hosts-file\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625512    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-cnibin\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-socket-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625584    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd683971-80ad-4673-9fd9-1e1fdd69cda4-host-slash\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625559    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-cnibin\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswgh\" (UniqueName: \"kubernetes.io/projected/742e70a2-c439-4669-b404-fb48d7c4a419-kube-api-access-bswgh\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-netns\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.626467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625670    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktwt\" (UniqueName: \"kubernetes.io/projected/375473e0-3d80-476e-8c4e-84205706af05-kube-api-access-tktwt\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625694    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76693f47-85b7-4686-aac5-1c24d6ba4627-tmp-dir\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625716    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-bin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625718    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-run-netns\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625804    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-kubelet\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj29l\" (UniqueName: \"kubernetes.io/projected/9154bc9f-fb46-4c80-a45c-a886603cddee-kube-api-access-wj29l\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625880    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-bin\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.625917    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-host\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626012    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-kubelet\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626036    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626063    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626080    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-etc-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-host\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd683971-80ad-4673-9fd9-1e1fdd69cda4-iptables-alerter-script\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-host\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626195    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-system-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626266    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-system-cni-dir\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626306    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-netns\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-netd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626382    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovn-node-metrics-cert\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626414    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5js\" (UniqueName: \"kubernetes.io/projected/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-kube-api-access-px5js\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626466    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626464    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626580    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-registration-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626583    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-etc-kubernetes\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a460fef0-8b06-44b2-b61c-1269fbb825db-agent-certs\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-kubelet\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626649    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-etc-kubernetes\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-bin\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626746    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-var-lib-kubelet\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626781    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-system-cni-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626848    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd683971-80ad-4673-9fd9-1e1fdd69cda4-iptables-alerter-script\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.627866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626858    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-var-lib-kubelet\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.626896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/742e70a2-c439-4669-b404-fb48d7c4a419-system-cni-dir\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627334    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627378    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-os-release\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627402    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-hostroot\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkh4p\" (UniqueName: \"kubernetes.io/projected/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-kube-api-access-xkh4p\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627459    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/b78a69bb-c901-48fb-afc4-335eacdd79ec-kube-api-access-t4cz8\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-socket-dir-parent\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-os-release\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a460fef0-8b06-44b2-b61c-1269fbb825db-konnectivity-ca\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627600    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-socket-dir-parent\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-slash\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-env-overrides\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.627824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-hostroot\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628041    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-systemd\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.628740 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628120    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-cni-binary-copy\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-multus\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628321    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-serviceca\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628325    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-systemd\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b78a69bb-c901-48fb-afc4-335eacdd79ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-kubelet-config\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-dbus\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-systemd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628528    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-tmp\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628611    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnfbq\" (UniqueName: \"kubernetes.io/projected/dd683971-80ad-4673-9fd9-1e1fdd69cda4-kube-api-access-vnfbq\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628743    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/742e70a2-c439-4669-b404-fb48d7c4a419-cni-binary-copy\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628833    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9154bc9f-fb46-4c80-a45c-a886603cddee-host-var-lib-cni-multus\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.628972    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-cni-binary-copy\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629058    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-daemon-config\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629170    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysconfig\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-kubernetes\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.629603 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-sys\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629390    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-sysconfig\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-lib-modules\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629583    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-script-lib\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629855    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-lib-modules\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.629919    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-etc-kubernetes\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.630100    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9154bc9f-fb46-4c80-a45c-a886603cddee-multus-daemon-config\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.630488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.630301    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-etc-tuned\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.631294 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.631270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/375473e0-3d80-476e-8c4e-84205706af05-tmp\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.632251 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.632226    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/375473e0-3d80-476e-8c4e-84205706af05-sys\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.639499 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.639464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kzw\" (UniqueName: \"kubernetes.io/projected/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-kube-api-access-55kzw\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:09.641096 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.641059    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnfbq\" (UniqueName: \"kubernetes.io/projected/dd683971-80ad-4673-9fd9-1e1fdd69cda4-kube-api-access-vnfbq\") pod \"iptables-alerter-f7fj6\" (UID: \"dd683971-80ad-4673-9fd9-1e1fdd69cda4\") " pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.641509 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.641438    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/b78a69bb-c901-48fb-afc4-335eacdd79ec-kube-api-access-t4cz8\") pod \"aws-ebs-csi-driver-node-4n5vf\" (UID: \"b78a69bb-c901-48fb-afc4-335eacdd79ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.641609 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.641544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktwt\" (UniqueName: \"kubernetes.io/projected/375473e0-3d80-476e-8c4e-84205706af05-kube-api-access-tktwt\") pod \"tuned-vh8lq\" (UID: \"375473e0-3d80-476e-8c4e-84205706af05\") " pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.641609 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.641544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswgh\" (UniqueName: \"kubernetes.io/projected/742e70a2-c439-4669-b404-fb48d7c4a419-kube-api-access-bswgh\") pod \"multus-additional-cni-plugins-xq7m7\" (UID: \"742e70a2-c439-4669-b404-fb48d7c4a419\") " pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.642393 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.642374    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfktq\" (UniqueName: \"kubernetes.io/projected/76693f47-85b7-4686-aac5-1c24d6ba4627-kube-api-access-zfktq\") pod \"node-resolver-rwsrm\" (UID: \"76693f47-85b7-4686-aac5-1c24d6ba4627\") " pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.642447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.642371    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj29l\" (UniqueName: \"kubernetes.io/projected/9154bc9f-fb46-4c80-a45c-a886603cddee-kube-api-access-wj29l\") pod \"multus-h45jj\" (UID: \"9154bc9f-fb46-4c80-a45c-a886603cddee\") " pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.726747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.726698    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:29:09.730135 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730109    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-kubelet-config\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.730257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730160    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-dbus\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.730257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730202    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-systemd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730234    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-dbus\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.730257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-script-lib\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730288    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-systemd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730301    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/299a9e33-f924-4aea-b3b0-15bde11145dc-kubelet-config\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-ovn\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730348    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-ovn\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-var-lib-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-var-lib-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-node-log\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730450    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730466    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-node-log\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730480    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-log-socket\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730511    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-run-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730518    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-systemd-units\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730546    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-log-socket\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730578    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-config\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730609    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-systemd-units\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-host\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730643    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:09.730710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730704    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-host\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-etc-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-etc-openvswitch\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.730820    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730870    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-netns\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-run-netns\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.730932    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:10.23086304 +0000 UTC m=+2.133091434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-netd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.730975    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-script-lib\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovn-node-metrics-cert\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5js\" (UniqueName: \"kubernetes.io/projected/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-kube-api-access-px5js\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a460fef0-8b06-44b2-b61c-1269fbb825db-agent-certs\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731097    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-kubelet\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731044    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-netd\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731124    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-bin\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731163    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovnkube-config\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731160    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkh4p\" (UniqueName: \"kubernetes.io/projected/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-kube-api-access-xkh4p\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.731531 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-kubelet\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731209    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a460fef0-8b06-44b2-b61c-1269fbb825db-konnectivity-ca\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731240    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-cni-bin\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-slash\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-host-slash\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731309    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-env-overrides\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731362    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-serviceca\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-env-overrides\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a460fef0-8b06-44b2-b61c-1269fbb825db-konnectivity-ca\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.732142 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.731853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-serviceca\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.734200 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.734172    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-ovn-node-metrics-cert\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.734200 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.734186    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a460fef0-8b06-44b2-b61c-1269fbb825db-agent-certs\") pod \"konnectivity-agent-979pr\" (UID: \"a460fef0-8b06-44b2-b61c-1269fbb825db\") " pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.741338 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.741311    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:09.741338 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.741341    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:09.741571 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.741357    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:09.741571 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:09.741449    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:10.241426437 +0000 UTC m=+2.143654814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:09.742665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.742636    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkh4p\" (UniqueName: \"kubernetes.io/projected/20587f8c-a3a2-48cf-b39d-a7a98a55c2d3-kube-api-access-xkh4p\") pod \"node-ca-f8mdt\" (UID: \"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3\") " pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.743694 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.743666    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5js\" (UniqueName: \"kubernetes.io/projected/1ef0b97e-f94b-497d-abb3-ab331d70b1cc-kube-api-access-px5js\") pod \"ovnkube-node-k49lk\" (UID: \"1ef0b97e-f94b-497d-abb3-ab331d70b1cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.830644 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.830568    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:29:09.842194 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.842158    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf"
Jul 29 10:29:09.859914 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.859879    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vh8lq"
Jul 29 10:29:09.874662 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.874629    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwsrm"
Jul 29 10:29:09.892362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.892317    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xq7m7"
Jul 29 10:29:09.909503 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.909469    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h45jj"
Jul 29 10:29:09.912639 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.912597    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25e3ac4c36569f4c5a0d44ba195e649.slice/crio-26eef490561d59245f62b03ec3ed749fa3de62a701d8032b06ca1772a8b5d9a1 WatchSource:0}: Error finding container 26eef490561d59245f62b03ec3ed749fa3de62a701d8032b06ca1772a8b5d9a1: Status 404 returned error can't find the container with id 26eef490561d59245f62b03ec3ed749fa3de62a701d8032b06ca1772a8b5d9a1
Jul 29 10:29:09.913353 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.913323    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742e70a2_c439_4669_b404_fb48d7c4a419.slice/crio-7f40ca3e764526c11f062133d4d9659cafb73ae03f3d0461f6e36440aa76ae72 WatchSource:0}: Error finding container 7f40ca3e764526c11f062133d4d9659cafb73ae03f3d0461f6e36440aa76ae72: Status 404 returned error can't find the container with id 7f40ca3e764526c11f062133d4d9659cafb73ae03f3d0461f6e36440aa76ae72
Jul 29 10:29:09.914022 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.913983    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76693f47_85b7_4686_aac5_1c24d6ba4627.slice/crio-b54a52d11662dd208cf1ce6e27b4d5757124dcf27ffb76a458c313d27f416d71 WatchSource:0}: Error finding container b54a52d11662dd208cf1ce6e27b4d5757124dcf27ffb76a458c313d27f416d71: Status 404 returned error can't find the container with id b54a52d11662dd208cf1ce6e27b4d5757124dcf27ffb76a458c313d27f416d71
Jul 29 10:29:09.920751 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.920707    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9154bc9f_fb46_4c80_a45c_a886603cddee.slice/crio-40f37a13f69d05970a25029d878ac5dd9f135e071e5fc1d4f0bbed0ad68634fb WatchSource:0}: Error finding container 40f37a13f69d05970a25029d878ac5dd9f135e071e5fc1d4f0bbed0ad68634fb: Status 404 returned error can't find the container with id 40f37a13f69d05970a25029d878ac5dd9f135e071e5fc1d4f0bbed0ad68634fb
Jul 29 10:29:09.922102 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.922047    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:29:09.925959 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.925927    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7fj6"
Jul 29 10:29:09.936721 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.936693    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd683971_80ad_4673_9fd9_1e1fdd69cda4.slice/crio-0790112eb18198072cae0437be5d29b3c4546eee75529ec163ce49b6cc2b1e08 WatchSource:0}: Error finding container 0790112eb18198072cae0437be5d29b3c4546eee75529ec163ce49b6cc2b1e08: Status 404 returned error can't find the container with id 0790112eb18198072cae0437be5d29b3c4546eee75529ec163ce49b6cc2b1e08
Jul 29 10:29:09.957849 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.957813    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:09.967161 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.967128    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda460fef0_8b06_44b2_b61c_1269fbb825db.slice/crio-7d27f4dc16dfe481eda642f302584f9bb65a6ca956531fab0865ada1b8becf90 WatchSource:0}: Error finding container 7d27f4dc16dfe481eda642f302584f9bb65a6ca956531fab0865ada1b8becf90: Status 404 returned error can't find the container with id 7d27f4dc16dfe481eda642f302584f9bb65a6ca956531fab0865ada1b8becf90
Jul 29 10:29:09.974597 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.974574    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8mdt"
Jul 29 10:29:09.979972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:09.979953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:09.985251 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.985219    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20587f8c_a3a2_48cf_b39d_a7a98a55c2d3.slice/crio-8d7d14937be04c273ddb3b756559a0c173319ce97c5ce0aaf1e64bf21302e28e WatchSource:0}: Error finding container 8d7d14937be04c273ddb3b756559a0c173319ce97c5ce0aaf1e64bf21302e28e: Status 404 returned error can't find the container with id 8d7d14937be04c273ddb3b756559a0c173319ce97c5ce0aaf1e64bf21302e28e
Jul 29 10:29:09.989810 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:09.989782    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef0b97e_f94b_497d_abb3_ab331d70b1cc.slice/crio-24cf7caf9849c86f52933928e4254e1767dc486a95fb7f24a017ba88d78c97af WatchSource:0}: Error finding container 24cf7caf9849c86f52933928e4254e1767dc486a95fb7f24a017ba88d78c97af: Status 404 returned error can't find the container with id 24cf7caf9849c86f52933928e4254e1767dc486a95fb7f24a017ba88d78c97af
Jul 29 10:29:10.133265 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.133162    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:10.133441 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.133314    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:10.133441 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.133373    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:11.133355137 +0000 UTC m=+3.035583517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:10.137466 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:10.137435    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45946e5e38ff9e43b00ea351a5826bd3.slice/crio-dd43c597f0f39cce9e47cb7e13114267707d018e50185efd604e1cd1af4ad43a WatchSource:0}: Error finding container dd43c597f0f39cce9e47cb7e13114267707d018e50185efd604e1cd1af4ad43a: Status 404 returned error can't find the container with id dd43c597f0f39cce9e47cb7e13114267707d018e50185efd604e1cd1af4ad43a
Jul 29 10:29:10.234420 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.234383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:10.234641 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.234506    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:10.234641 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.234588    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:11.23456408 +0000 UTC m=+3.136792455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:10.327708 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:10.327668    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375473e0_3d80_476e_8c4e_84205706af05.slice/crio-53564e64da3e553d0891441d424cb25dcb6421cecb154361f85a9f2f29bf44a4 WatchSource:0}: Error finding container 53564e64da3e553d0891441d424cb25dcb6421cecb154361f85a9f2f29bf44a4: Status 404 returned error can't find the container with id 53564e64da3e553d0891441d424cb25dcb6421cecb154361f85a9f2f29bf44a4
Jul 29 10:29:10.335133 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.335103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:10.335322 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.335295    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:10.335322 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.335326    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:10.335537 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.335343    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:10.335537 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.335413    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:11.335392688 +0000 UTC m=+3.237621088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:10.344336 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.344300    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:29:10.565119 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.564896    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:24:10 +0000 UTC" deadline="2028-01-28 15:54:52.941786573 +0000 UTC"
Jul 29 10:29:10.565119 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.564956    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13157h25m42.376835105s"
Jul 29 10:29:10.723615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.723576    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:10.723822 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:10.723733    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:10.730786 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.730721    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-979pr" event={"ID":"a460fef0-8b06-44b2-b61c-1269fbb825db","Type":"ContainerStarted","Data":"7d27f4dc16dfe481eda642f302584f9bb65a6ca956531fab0865ada1b8becf90"}
Jul 29 10:29:10.742818 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.742765    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7fj6" event={"ID":"dd683971-80ad-4673-9fd9-1e1fdd69cda4","Type":"ContainerStarted","Data":"0790112eb18198072cae0437be5d29b3c4546eee75529ec163ce49b6cc2b1e08"}
Jul 29 10:29:10.745483 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.745398    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h45jj" event={"ID":"9154bc9f-fb46-4c80-a45c-a886603cddee","Type":"ContainerStarted","Data":"40f37a13f69d05970a25029d878ac5dd9f135e071e5fc1d4f0bbed0ad68634fb"}
Jul 29 10:29:10.753767 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.753659    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwsrm" event={"ID":"76693f47-85b7-4686-aac5-1c24d6ba4627","Type":"ContainerStarted","Data":"b54a52d11662dd208cf1ce6e27b4d5757124dcf27ffb76a458c313d27f416d71"}
Jul 29 10:29:10.764355 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.764306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh8lq" event={"ID":"375473e0-3d80-476e-8c4e-84205706af05","Type":"ContainerStarted","Data":"53564e64da3e553d0891441d424cb25dcb6421cecb154361f85a9f2f29bf44a4"}
Jul 29 10:29:10.773766 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.773538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal" event={"ID":"45946e5e38ff9e43b00ea351a5826bd3","Type":"ContainerStarted","Data":"dd43c597f0f39cce9e47cb7e13114267707d018e50185efd604e1cd1af4ad43a"}
Jul 29 10:29:10.783402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.783356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"24cf7caf9849c86f52933928e4254e1767dc486a95fb7f24a017ba88d78c97af"}
Jul 29 10:29:10.794688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.794637    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8mdt" event={"ID":"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3","Type":"ContainerStarted","Data":"8d7d14937be04c273ddb3b756559a0c173319ce97c5ce0aaf1e64bf21302e28e"}
Jul 29 10:29:10.800146 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.800099    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerStarted","Data":"7f40ca3e764526c11f062133d4d9659cafb73ae03f3d0461f6e36440aa76ae72"}
Jul 29 10:29:10.818641 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:10.818440    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal" event={"ID":"e25e3ac4c36569f4c5a0d44ba195e649","Type":"ContainerStarted","Data":"26eef490561d59245f62b03ec3ed749fa3de62a701d8032b06ca1772a8b5d9a1"}
Jul 29 10:29:11.144282 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.144180    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:11.144486 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.144412    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:11.144486 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.144486    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:13.144464387 +0000 UTC m=+5.046692789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:11.245334 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.245289    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:11.245530 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.245468    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:11.245657 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.245537    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:13.245516763 +0000 UTC m=+5.147745139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:11.346155 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.346093    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:11.346411 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.346324    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:11.346411 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.346351    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:11.346411 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.346367    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:11.346596 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.346435    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:13.346414526 +0000 UTC m=+5.248642906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:11.565416 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.565364    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:24:10 +0000 UTC" deadline="2027-12-26 18:59:02.03866273 +0000 UTC"
Jul 29 10:29:11.565416 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.565414    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12368h29m50.473254562s"
Jul 29 10:29:11.723676 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.723639    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:11.723895 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.723806    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:11.724299 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.724278    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:11.724421 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:11.724385    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:11.841518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:11.841376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf" event={"ID":"b78a69bb-c901-48fb-afc4-335eacdd79ec","Type":"ContainerStarted","Data":"a4736a35137d71857dc010e2ba0a4e40d712944060e7b34fb64c43020c2d2ad0"}
Jul 29 10:29:12.732473 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:12.732365    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:12.732982 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:12.732579    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:13.163436 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:13.163336    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:13.163630 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.163559    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:13.163733 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.163635    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.163612922 +0000 UTC m=+9.065841314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:13.264602 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:13.264556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:13.264897 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.264728    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:13.264897 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.264800    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.264781042 +0000 UTC m=+9.167009440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:13.365892 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:13.365305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:13.365892 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.365487    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:13.365892 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.365507    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:13.365892 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.365521    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:13.365892 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.365578    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.365562019 +0000 UTC m=+9.267790407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:13.723202 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:13.723162    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:13.723490 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.723355    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:13.723815 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:13.723792    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:13.723931 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:13.723901    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:14.726320 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:14.726281    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:14.726851 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:14.726494    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:15.723354 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:15.723114    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:15.723354 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:15.723288    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:15.723354 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:15.723352    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:15.723698 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:15.723489    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:16.723781 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:16.723487    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:16.724329 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:16.723897    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:17.201171 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:17.201020    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:17.201355 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.201218    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.201355 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.201294    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:25.201271535 +0000 UTC m=+17.103499927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.301722 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:17.301572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:17.301929 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.301752    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.301929 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.301829    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:25.301808621 +0000 UTC m=+17.204036997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.402630 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:17.402577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:17.402847 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.402816    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:17.402847 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.402839    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:17.402959 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.402852    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:17.402959 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.402920    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:25.402899741 +0000 UTC m=+17.305128135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:17.723615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:17.723582    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:17.723817 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.723727    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:17.724257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:17.724163    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:17.724319 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:17.724259    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:18.724597 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:18.723993    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:18.724597 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:18.724144    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:19.722934 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:19.722885    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:19.723099 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:19.723067    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:19.723249 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:19.723227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:19.723375 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:19.723352    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:20.722874 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:20.722835    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:20.723282 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:20.722964    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:21.723437 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:21.723394    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:21.723437 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:21.723430    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:21.723948 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:21.723553    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:21.723948 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:21.723656    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:22.722982 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:22.722944    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:22.723203 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:22.723082    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:23.723425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:23.723382    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:23.723425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:23.723414    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:23.723925 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:23.723553    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:23.723925 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:23.723679    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:24.723506 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:24.723466    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:24.723998 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:24.723620    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:25.269687 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:25.269642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:25.269900 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.269798    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:25.269900 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.269868    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:41.269850296 +0000 UTC m=+33.172078671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:25.370638 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:25.370601    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:25.370858 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.370772    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:25.370858 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.370855    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:29:41.37083333 +0000 UTC m=+33.273061723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:25.471015 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:25.470964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:25.471227 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.471102    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:25.471227 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.471138    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:25.471227 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.471151    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:25.471227 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.471210    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:41.471190647 +0000 UTC m=+33.373419043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:25.723157 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:25.723055    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:25.723157 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:25.723114    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:25.723428 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.723221    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:25.723428 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:25.723314    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:26.723264 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:26.723223    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:26.723659 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:26.723366    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:27.723589 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:27.723541    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:27.724099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:27.723563    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:27.724099 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:27.723716    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:27.724099 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:27.723793    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:28.724076 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:28.724039    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:28.724551 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:28.724182    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:29.723440 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:29.723402    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:29.723644 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:29.723411    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:29.723644 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:29.723542    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:29.723644 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:29.723615    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:30.723149 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.723096    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:30.723554 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:30.723229    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:30.895693 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.895655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"9fbb0ae08573ffdca2619a9daad878e98b5ff74b3dbc59f8a8a9f292a0d486b2"}
Jul 29 10:29:30.898392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.898359    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal" event={"ID":"e25e3ac4c36569f4c5a0d44ba195e649","Type":"ContainerStarted","Data":"ed0bc2895c2c5bace734bc2a473904e4115f2ec7eab9cc37c827cc89178c7918"}
Jul 29 10:29:30.899801 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.899769    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vh8lq" event={"ID":"375473e0-3d80-476e-8c4e-84205706af05","Type":"ContainerStarted","Data":"fa74aa42e1a18ae8b47e6961345fb7ad8e670cf18cf98edd0249b3621d7799a2"}
Jul 29 10:29:30.916776 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.916715    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-91.ec2.internal" podStartSLOduration=20.916692458 podStartE2EDuration="20.916692458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:30.915659433 +0000 UTC m=+22.817887833" watchObservedRunningTime="2026-07-29 10:29:30.916692458 +0000 UTC m=+22.818920856"
Jul 29 10:29:30.936919 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:30.936220    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vh8lq" podStartSLOduration=1.682873556 podStartE2EDuration="21.936194456s" totalImagesPullingTime="20.2533209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:30.935460421 +0000 UTC m=+22.837688824" watchObservedRunningTime="2026-07-29 10:29:30.936194456 +0000 UTC m=+22.838422854"
Jul 29 10:29:31.723595 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.723261    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:31.724332 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.723277    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:31.724332 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:31.723711    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:31.724332 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:31.723735    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:31.903872 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.903800    2551 generic.go:356] "Generic (PLEG): container finished" podID="45946e5e38ff9e43b00ea351a5826bd3" containerID="0e9435d5afec5d0ee2c9ebc1e02b1bf00497751c9dec5c4afa3bbc8a6c24f364" exitCode=0
Jul 29 10:29:31.904044 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.903876    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal" event={"ID":"45946e5e38ff9e43b00ea351a5826bd3","Type":"ContainerDied","Data":"0e9435d5afec5d0ee2c9ebc1e02b1bf00497751c9dec5c4afa3bbc8a6c24f364"}
Jul 29 10:29:31.906733 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.906713    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:29:31.907032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907011    2551 generic.go:356] "Generic (PLEG): container finished" podID="1ef0b97e-f94b-497d-abb3-ab331d70b1cc" containerID="d789a17e929734944da2fa7d1dba3fde46d63292b0972397ba0bcaf81e05ca92" exitCode=1
Jul 29 10:29:31.907113 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907079    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"810e4d29a0495ef39f8b0c63de35e057785b03bd0098f83cbdf29721ef6c53ad"}
Jul 29 10:29:31.907210 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907196    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"ca7653f671c0263e807ae4421669206b8bdca1fbdd239d41b8576503a7ba645d"}
Jul 29 10:29:31.907252 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907235    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"4a5e439b8b35b1f6539e319850b35b6b8e0357e84c6f7399ba71c14b1640fba4"}
Jul 29 10:29:31.907286 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907265    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"1184f536e5a680508f33ca535bed6e9d69287cb78b2e4a09e7aaf062aae56328"}
Jul 29 10:29:31.907321 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.907294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerDied","Data":"d789a17e929734944da2fa7d1dba3fde46d63292b0972397ba0bcaf81e05ca92"}
Jul 29 10:29:31.909084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.909041    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8mdt" event={"ID":"20587f8c-a3a2-48cf-b39d-a7a98a55c2d3","Type":"ContainerStarted","Data":"66d5e4ede696bb226565afa0ea4a70bab220fae916f5e5ac673671f0f4f4c101"}
Jul 29 10:29:31.910676 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.910654    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="aadc9ac69c7f5eb656225f629d017e1034eb365df757bdba37f42eda847131f6" exitCode=0
Jul 29 10:29:31.910770 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.910723    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"aadc9ac69c7f5eb656225f629d017e1034eb365df757bdba37f42eda847131f6"}
Jul 29 10:29:31.911981 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.911958    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-979pr" event={"ID":"a460fef0-8b06-44b2-b61c-1269fbb825db","Type":"ContainerStarted","Data":"27dfa6dd4a82022add4d620eeac4ab37829b2a43bbb839cd1a213c27ca03826e"}
Jul 29 10:29:31.913559 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.913534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h45jj" event={"ID":"9154bc9f-fb46-4c80-a45c-a886603cddee","Type":"ContainerStarted","Data":"1ec5d211ba4d70993b0961a2f4111cd4944b8c0360572e284592fa0aaa0a2f76"}
Jul 29 10:29:31.914893 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.914874    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwsrm" event={"ID":"76693f47-85b7-4686-aac5-1c24d6ba4627","Type":"ContainerStarted","Data":"478ad6cf3f22a6e2b446103ed2259636f0777d3c5f5cf0d4786319881a240cc0"}
Jul 29 10:29:31.916239 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.916221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf" event={"ID":"b78a69bb-c901-48fb-afc4-335eacdd79ec","Type":"ContainerStarted","Data":"c781845e9ab1df78f8b7b75afedeaed270679e19332d76933e65f767f9d3e3a0"}
Jul 29 10:29:31.942357 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.941943    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-979pr" podStartSLOduration=2.436091437 podStartE2EDuration="22.941921664s" totalImagesPullingTime="20.505830227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:31.941339385 +0000 UTC m=+23.843567794" watchObservedRunningTime="2026-07-29 10:29:31.941921664 +0000 UTC m=+23.844150063"
Jul 29 10:29:31.959041 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.958980    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f8mdt" podStartSLOduration=6.162372664 podStartE2EDuration="22.958959227s" totalImagesPullingTime="16.796586563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:31.958450393 +0000 UTC m=+23.860678791" watchObservedRunningTime="2026-07-29 10:29:31.958959227 +0000 UTC m=+23.861187626"
Jul 29 10:29:31.975814 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:31.975753    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rwsrm" podStartSLOduration=2.305113511 podStartE2EDuration="22.975734555s" totalImagesPullingTime="20.670621044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:31.975293101 +0000 UTC m=+23.877521498" watchObservedRunningTime="2026-07-29 10:29:31.975734555 +0000 UTC m=+23.877962951"
Jul 29 10:29:32.019218 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.019166    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h45jj" podStartSLOduration=2.037246501 podStartE2EDuration="23.019112878s" totalImagesPullingTime="20.981866377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:32.018650671 +0000 UTC m=+23.920879069" watchObservedRunningTime="2026-07-29 10:29:32.019112878 +0000 UTC m=+23.921341274"
Jul 29 10:29:32.611375 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.611309    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:32.723437 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.723349    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:32.723627 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:32.723491    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:32.921975 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.921440    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7fj6" event={"ID":"dd683971-80ad-4673-9fd9-1e1fdd69cda4","Type":"ContainerStarted","Data":"482ed068c33edf3763b087fb9683b58417ba16f54a96358f613344d55978d896"}
Jul 29 10:29:32.924194 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.924160    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf" event={"ID":"b78a69bb-c901-48fb-afc4-335eacdd79ec","Type":"ContainerStarted","Data":"220b116cab560e56c956bf7da9aa5ea69dc02af3ae6234a1381517b44988f3aa"}
Jul 29 10:29:32.926462 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.926429    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal" event={"ID":"45946e5e38ff9e43b00ea351a5826bd3","Type":"ContainerStarted","Data":"a7b59c8ed75e9ae8acea526c490d5b692dbc286df0372b5307d09081cdb0f492"}
Jul 29 10:29:32.938270 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.938209    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f7fj6" podStartSLOduration=3.295107842 podStartE2EDuration="23.938186802s" totalImagesPullingTime="20.64307896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:32.937578665 +0000 UTC m=+24.839807063" watchObservedRunningTime="2026-07-29 10:29:32.938186802 +0000 UTC m=+24.840415204"
Jul 29 10:29:32.956061 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:32.955994    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-91.ec2.internal" podStartSLOduration=22.955969393 podStartE2EDuration="22.955969393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:32.955468535 +0000 UTC m=+24.857696931" watchObservedRunningTime="2026-07-29 10:29:32.955969393 +0000 UTC m=+24.858197792"
Jul 29 10:29:33.590013 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.589914    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:32.611346878Z","UUID":"502af52a-d177-4405-a93a-c9a0ae2fa249","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:33.591916 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.591889    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:33.592049 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.591971    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:33.723643 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.723605    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:33.724095 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.723624    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:33.724095 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:33.723786    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:33.724095 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:33.723860    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:33.931659 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.931559    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf" event={"ID":"b78a69bb-c901-48fb-afc4-335eacdd79ec","Type":"ContainerStarted","Data":"ebf270aab3d7c5bf08fe4d14a50bb02d4bddc5c24ef05f3b897c9abddbabc995"}
Jul 29 10:29:33.935221 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.935198    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:29:33.935729 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:33.935677    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"ca1220b8fd855f185bfd1393a4ef18b337385c2a4784636df95ff227a9f8aac5"}
Jul 29 10:29:34.723148 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:34.723101    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:34.723459 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:34.723250    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:35.445517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.442741    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:35.446485 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.446454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:35.461650 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.461583    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4n5vf" podStartSLOduration=4.460685669 podStartE2EDuration="26.461556869s" totalImagesPullingTime="22.0008712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:33.95252685 +0000 UTC m=+25.854755248" watchObservedRunningTime="2026-07-29 10:29:35.461556869 +0000 UTC m=+27.363785263"
Jul 29 10:29:35.723304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.723203    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:35.723304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.723227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:35.723550 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:35.723352    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:35.723550 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:35.723518    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:35.940212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.940178    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:35.940801 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:35.940764    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-979pr"
Jul 29 10:29:36.723454 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.723201    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:36.724099 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:36.723503    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:36.945555 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.945527    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:29:36.945934 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.945903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"6aa61631b29f3ecf1a990cad47cf00322a2dfafa5571705b7639d364baaa6507"}
Jul 29 10:29:36.946200 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.946184    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:36.946376 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.946360    2551 scope.go:117] "RemoveContainer" containerID="d789a17e929734944da2fa7d1dba3fde46d63292b0972397ba0bcaf81e05ca92"
Jul 29 10:29:36.948089 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.948061    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="7e6d05ed1bfefc9c68649f38389c968e9482bf3f3b6b4da8d05ed75176ae11a8" exitCode=0
Jul 29 10:29:36.948176 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.948157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"7e6d05ed1bfefc9c68649f38389c968e9482bf3f3b6b4da8d05ed75176ae11a8"}
Jul 29 10:29:36.963471 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:36.963446    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:37.723235 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.723199    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:37.723457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.723195    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:37.723457 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:37.723330    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:37.723457 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:37.723386    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:37.954327 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.954299    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:29:37.954706 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.954683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" event={"ID":"1ef0b97e-f94b-497d-abb3-ab331d70b1cc","Type":"ContainerStarted","Data":"972a1f765fb29cd1571ef2bc87b6d664efdf5518fcd14c15ae28fe68eec59554"}
Jul 29 10:29:37.954968 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.954952    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:37.955054 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.955001    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:37.970565 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.970537    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:29:37.991392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:37.991297    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk" podStartSLOduration=8.230454633 podStartE2EDuration="28.991280101s" totalImagesPullingTime="20.760825468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:37.989852968 +0000 UTC m=+29.892081366" watchObservedRunningTime="2026-07-29 10:29:37.991280101 +0000 UTC m=+29.893508497"
Jul 29 10:29:38.723609 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:38.723529    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:38.723996 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:38.723630    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:38.958911 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:38.958875    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="cd83a533447ecd33223481edff78069ee55240b336739031cb6312c1d6296e8f" exitCode=0
Jul 29 10:29:38.959099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:38.958961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"cd83a533447ecd33223481edff78069ee55240b336739031cb6312c1d6296e8f"}
Jul 29 10:29:39.722778 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:39.722738    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:39.722778 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:39.722771    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:39.723033 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:39.722903    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:39.723099 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:39.723037    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:40.723198 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:40.722989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:40.723593 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:40.723290    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:40.966051 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:40.966015    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="3c8a64f238a4bd085032a54aa157607beb8f100ce9b94fa471c67a5505171f72" exitCode=0
Jul 29 10:29:40.966236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:40.966064    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"3c8a64f238a4bd085032a54aa157607beb8f100ce9b94fa471c67a5505171f72"}
Jul 29 10:29:41.289297 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:41.289258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:41.289490 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.289428    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:41.289532 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.289501    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs podName:de60eff5-1df5-4c1b-b19b-3ca6c0f2479a nodeName:}" failed. No retries permitted until 2026-07-29 10:30:13.289484228 +0000 UTC m=+65.191712620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs") pod "network-metrics-daemon-9tsjs" (UID: "de60eff5-1df5-4c1b-b19b-3ca6c0f2479a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:41.389975 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:41.389930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:41.390170 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.390094    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:41.390208 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.390185    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret podName:299a9e33-f924-4aea-b3b0-15bde11145dc nodeName:}" failed. No retries permitted until 2026-07-29 10:30:13.390163119 +0000 UTC m=+65.292391513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret") pod "global-pull-secret-syncer-p5v7t" (UID: "299a9e33-f924-4aea-b3b0-15bde11145dc") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:41.490595 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:41.490550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:41.490793 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.490744    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:41.490793 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.490769    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:41.490793 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.490782    2551 projected.go:196] Error preparing data for projected volume kube-api-access-nnj8m for pod openshift-network-diagnostics/network-check-target-89sxt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:41.490950 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.490856    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m podName:df2f6da7-779a-4c19-a683-7d97b8d6eac2 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:13.490834684 +0000 UTC m=+65.393063079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnj8m" (UniqueName: "kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m") pod "network-check-target-89sxt" (UID: "df2f6da7-779a-4c19-a683-7d97b8d6eac2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:41.723561 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:41.723479    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:41.723561 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:41.723521    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:41.724085 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.723591    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:41.724085 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:41.723653    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:42.723209 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:42.723164    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:42.723412 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:42.723286    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:43.723640 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:43.723598    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:43.724079 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:43.723614    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:43.724079 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:43.723750    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:43.724079 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:43.723795    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:44.723237 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:44.723194    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:44.723417 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:44.723352    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:45.723020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:45.722979    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:45.723409 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:45.723098    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:45.723409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:45.723157    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:45.723409 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:45.723250    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:45.980838 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:45.980740    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="6ec04447af9f2baa6074205b2e443c6b7752d5ea22cb812f07cb4e9459036229" exitCode=0
Jul 29 10:29:45.980838 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:45.980785    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"6ec04447af9f2baa6074205b2e443c6b7752d5ea22cb812f07cb4e9459036229"}
Jul 29 10:29:46.723626 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:46.723583    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:46.724046 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:46.723712    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:46.986496 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:46.986405    2551 generic.go:356] "Generic (PLEG): container finished" podID="742e70a2-c439-4669-b404-fb48d7c4a419" containerID="15f1a6c8c6b2c2149597613e7bd66ab086c3651e5cf3ec4a8349936a6e5c118b" exitCode=0
Jul 29 10:29:46.986496 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:46.986470    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerDied","Data":"15f1a6c8c6b2c2149597613e7bd66ab086c3651e5cf3ec4a8349936a6e5c118b"}
Jul 29 10:29:47.722918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:47.722879    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:47.723097 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:47.722982    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:47.723097 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:47.723043    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:47.723256 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:47.723179    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:47.992748 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:47.992656    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" event={"ID":"742e70a2-c439-4669-b404-fb48d7c4a419","Type":"ContainerStarted","Data":"369413d9fc85db9553241a9d16af553a9f16b0047844d9ab0f11415cd6db79fb"}
Jul 29 10:29:48.016032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:48.015975    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xq7m7" podStartSLOduration=7.200893115 podStartE2EDuration="39.01595615s" totalImagesPullingTime="31.815063035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:48.015328315 +0000 UTC m=+39.917556716" watchObservedRunningTime="2026-07-29 10:29:48.01595615 +0000 UTC m=+39.918184546"
Jul 29 10:29:48.725700 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:48.725657    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:48.725963 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:48.725757    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:49.462965 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.462762    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5v7t"]
Jul 29 10:29:49.463480 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.463038    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:49.463480 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:49.463165    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:49.465979 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.465943    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89sxt"]
Jul 29 10:29:49.466683 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.466659    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:49.466817 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.466684    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tsjs"]
Jul 29 10:29:49.466817 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:49.466782    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:49.466935 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:49.466817    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:49.467109 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:49.467005    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:50.723725 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:50.723629    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:50.724151 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:50.723631    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:50.724151 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:50.723746    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:50.724151 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:50.723827    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:51.723265 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:51.723231    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:51.723464 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:51.723338    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:52.723059 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:52.723014    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:52.723059 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:52.723058    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:52.723516 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:52.723169    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89sxt" podUID="df2f6da7-779a-4c19-a683-7d97b8d6eac2"
Jul 29 10:29:52.723516 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:52.723261    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5v7t" podUID="299a9e33-f924-4aea-b3b0-15bde11145dc"
Jul 29 10:29:53.722704 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.722675    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:53.722876 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:53.722842    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tsjs" podUID="de60eff5-1df5-4c1b-b19b-3ca6c0f2479a"
Jul 29 10:29:53.846162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.846111    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-91.ec2.internal" event="NodeReady"
Jul 29 10:29:53.846532 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.846269    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:53.895611 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.895577    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-49kqc"]
Jul 29 10:29:53.921845 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.921814    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s5cqz"]
Jul 29 10:29:53.922013 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.921969    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:53.924236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.924206    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:53.924495 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.924478    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:53.924556 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.924479    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4pw9d\""
Jul 29 10:29:53.939874 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.939838    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s5cqz"]
Jul 29 10:29:53.940040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.939883    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49kqc"]
Jul 29 10:29:53.940040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.939910    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ccdw4"]
Jul 29 10:29:53.940040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.939963    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:53.942723 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.942699    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:53.943223 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.943204    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pbtf2\""
Jul 29 10:29:53.943491 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.943473    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:53.943867 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.943848    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:53.956002 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.955970    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccdw4"]
Jul 29 10:29:53.956168 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.956123    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:53.959231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.959204    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:29:53.959860 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.959842    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:29:53.959980 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.959893    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:29:53.960043 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.960000    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jl44f\""
Jul 29 10:29:53.960108 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:53.960070    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:29:54.082158 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80612ed9-585a-4273-9330-304fada0fe36-metrics-tls\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.082158 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082084    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8e611c33-c2c7-49f5-965e-116fa6ae10e4-crio-socket\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082180    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-kube-api-access-dz6wv\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082207    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082232    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8e611c33-c2c7-49f5-965e-116fa6ae10e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082266    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswlr\" (UniqueName: \"kubernetes.io/projected/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-api-access-wswlr\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082299    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8mt\" (UniqueName: \"kubernetes.io/projected/80612ed9-585a-4273-9330-304fada0fe36-kube-api-access-bf8mt\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082322    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80612ed9-585a-4273-9330-304fada0fe36-config-volume\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082337    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-cert\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.082370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e611c33-c2c7-49f5-965e-116fa6ae10e4-data-volume\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.082621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.082397    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80612ed9-585a-4273-9330-304fada0fe36-tmp-dir\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.182956 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.182916    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswlr\" (UniqueName: \"kubernetes.io/projected/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-api-access-wswlr\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.182977    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8mt\" (UniqueName: \"kubernetes.io/projected/80612ed9-585a-4273-9330-304fada0fe36-kube-api-access-bf8mt\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183009    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80612ed9-585a-4273-9330-304fada0fe36-config-volume\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-cert\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183051    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e611c33-c2c7-49f5-965e-116fa6ae10e4-data-volume\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183110    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80612ed9-585a-4273-9330-304fada0fe36-tmp-dir\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.183190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80612ed9-585a-4273-9330-304fada0fe36-metrics-tls\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183195    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8e611c33-c2c7-49f5-965e-116fa6ae10e4-crio-socket\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183226    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-kube-api-access-dz6wv\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183278    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8e611c33-c2c7-49f5-965e-116fa6ae10e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8e611c33-c2c7-49f5-965e-116fa6ae10e4-crio-socket\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183505    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8e611c33-c2c7-49f5-965e-116fa6ae10e4-data-volume\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183724    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80612ed9-585a-4273-9330-304fada0fe36-config-volume\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.183950 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.183950 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.183938    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80612ed9-585a-4273-9330-304fada0fe36-tmp-dir\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.187489 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.187464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80612ed9-585a-4273-9330-304fada0fe36-metrics-tls\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.187620 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.187466    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8e611c33-c2c7-49f5-965e-116fa6ae10e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.187620 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.187499    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-cert\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.195404 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.195374    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8mt\" (UniqueName: \"kubernetes.io/projected/80612ed9-585a-4273-9330-304fada0fe36-kube-api-access-bf8mt\") pod \"dns-default-49kqc\" (UID: \"80612ed9-585a-4273-9330-304fada0fe36\") " pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.203734 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.203703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswlr\" (UniqueName: \"kubernetes.io/projected/8e611c33-c2c7-49f5-965e-116fa6ae10e4-kube-api-access-wswlr\") pod \"insights-runtime-extractor-ccdw4\" (UID: \"8e611c33-c2c7-49f5-965e-116fa6ae10e4\") " pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.208850 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.208823    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6wv\" (UniqueName: \"kubernetes.io/projected/5b0ba4af-fd88-4838-ae7a-2ce433e29b4a-kube-api-access-dz6wv\") pod \"ingress-canary-s5cqz\" (UID: \"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a\") " pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.234660 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.234627    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-49kqc"
Jul 29 10:29:54.248803 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.248765    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s5cqz"
Jul 29 10:29:54.265096 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.265074    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccdw4"
Jul 29 10:29:54.485178 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.485122    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s5cqz"]
Jul 29 10:29:54.488788 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.488757    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccdw4"]
Jul 29 10:29:54.490173 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:54.490098    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0ba4af_fd88_4838_ae7a_2ce433e29b4a.slice/crio-b0ed124aab5615d2723c0b5e6d953024c0ecc025b66c93748a2c72377a80777f WatchSource:0}: Error finding container b0ed124aab5615d2723c0b5e6d953024c0ecc025b66c93748a2c72377a80777f: Status 404 returned error can't find the container with id b0ed124aab5615d2723c0b5e6d953024c0ecc025b66c93748a2c72377a80777f
Jul 29 10:29:54.492349 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:54.492328    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e611c33_c2c7_49f5_965e_116fa6ae10e4.slice/crio-3e676ee5d46b36f6cc8d764840ba0999f7e3b22bbd5d61c518ae6b4d2eddd39a WatchSource:0}: Error finding container 3e676ee5d46b36f6cc8d764840ba0999f7e3b22bbd5d61c518ae6b4d2eddd39a: Status 404 returned error can't find the container with id 3e676ee5d46b36f6cc8d764840ba0999f7e3b22bbd5d61c518ae6b4d2eddd39a
Jul 29 10:29:54.496370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.496347    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-49kqc"]
Jul 29 10:29:54.499711 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:29:54.499683    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80612ed9_585a_4273_9330_304fada0fe36.slice/crio-14ccdaf3d11c3247704e9e14f7994d65b8d1b4b85c0203a589db288ea3b4d949 WatchSource:0}: Error finding container 14ccdaf3d11c3247704e9e14f7994d65b8d1b4b85c0203a589db288ea3b4d949: Status 404 returned error can't find the container with id 14ccdaf3d11c3247704e9e14f7994d65b8d1b4b85c0203a589db288ea3b4d949
Jul 29 10:29:54.723593 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.723556    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:29:54.723792 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.723556    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:29:54.726692 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.726666    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9npp\""
Jul 29 10:29:54.726857 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.726714    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:54.727798 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.727779    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:54.727879 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:54.727789    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:55.011111 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.011016    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccdw4" event={"ID":"8e611c33-c2c7-49f5-965e-116fa6ae10e4","Type":"ContainerStarted","Data":"3e676ee5d46b36f6cc8d764840ba0999f7e3b22bbd5d61c518ae6b4d2eddd39a"}
Jul 29 10:29:55.012302 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.012272    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49kqc" event={"ID":"80612ed9-585a-4273-9330-304fada0fe36","Type":"ContainerStarted","Data":"14ccdaf3d11c3247704e9e14f7994d65b8d1b4b85c0203a589db288ea3b4d949"}
Jul 29 10:29:55.013659 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.013631    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s5cqz" event={"ID":"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a","Type":"ContainerStarted","Data":"b0ed124aab5615d2723c0b5e6d953024c0ecc025b66c93748a2c72377a80777f"}
Jul 29 10:29:55.723213 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.723164    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:29:55.727604 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.727557    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:55.727907 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:55.727887    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg7mn\""
Jul 29 10:29:56.557492 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.557186    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"]
Jul 29 10:29:56.577055 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.577006    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"]
Jul 29 10:29:56.577261 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.577187    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:56.581203 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.581168    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7rfwk\""
Jul 29 10:29:56.581596 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.581556    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 10:29:56.704147 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.704099    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9248b\" (UID: \"b09bf49f-251c-4e50-9a24-6e711b73e16f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:56.804654 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:56.804550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9248b\" (UID: \"b09bf49f-251c-4e50-9a24-6e711b73e16f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:56.804883 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:56.804679    2551 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:29:56.804883 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:56.804760    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates podName:b09bf49f-251c-4e50-9a24-6e711b73e16f nodeName:}" failed. No retries permitted until 2026-07-29 10:29:57.304738361 +0000 UTC m=+49.206966747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-9248b" (UID: "b09bf49f-251c-4e50-9a24-6e711b73e16f") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:29:57.308932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:57.308894    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9248b\" (UID: \"b09bf49f-251c-4e50-9a24-6e711b73e16f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:57.309259 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:57.309086    2551 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:29:57.309259 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:29:57.309210    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates podName:b09bf49f-251c-4e50-9a24-6e711b73e16f nodeName:}" failed. No retries permitted until 2026-07-29 10:29:58.309185357 +0000 UTC m=+50.211413745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-9248b" (UID: "b09bf49f-251c-4e50-9a24-6e711b73e16f") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:29:58.317944 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:58.317898    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9248b\" (UID: \"b09bf49f-251c-4e50-9a24-6e711b73e16f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:58.322090 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:58.322058    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b09bf49f-251c-4e50-9a24-6e711b73e16f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9248b\" (UID: \"b09bf49f-251c-4e50-9a24-6e711b73e16f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:29:58.391061 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:29:58.391021    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:30:00.649173 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:00.647504    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"]
Jul 29 10:30:00.655558 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:00.655492    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09bf49f_251c_4e50_9a24_6e711b73e16f.slice/crio-53d7a2de6453b7ce352d816f19b59dace9057a2f80cbcc39af570151a1ab2061 WatchSource:0}: Error finding container 53d7a2de6453b7ce352d816f19b59dace9057a2f80cbcc39af570151a1ab2061: Status 404 returned error can't find the container with id 53d7a2de6453b7ce352d816f19b59dace9057a2f80cbcc39af570151a1ab2061
Jul 29 10:30:01.032871 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.032832    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccdw4" event={"ID":"8e611c33-c2c7-49f5-965e-116fa6ae10e4","Type":"ContainerStarted","Data":"1164b8d43d58eddf70c16ea9b72f7fd117abf94846b450af5e1bde211e30581f"}
Jul 29 10:30:01.034568 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.034538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49kqc" event={"ID":"80612ed9-585a-4273-9330-304fada0fe36","Type":"ContainerStarted","Data":"de1299316b6358094bc6cc3b62776f36c9c97e3fbcaa96b10db2d116d2690145"}
Jul 29 10:30:01.034741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.034588    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-49kqc" event={"ID":"80612ed9-585a-4273-9330-304fada0fe36","Type":"ContainerStarted","Data":"8053e637fcff0852cba2c55f5f38340588ad887473685892fe84adf60e198d70"}
Jul 29 10:30:01.034741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.034690    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-49kqc"
Jul 29 10:30:01.035726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.035700    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b" event={"ID":"b09bf49f-251c-4e50-9a24-6e711b73e16f","Type":"ContainerStarted","Data":"53d7a2de6453b7ce352d816f19b59dace9057a2f80cbcc39af570151a1ab2061"}
Jul 29 10:30:01.037089 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.037064    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s5cqz" event={"ID":"5b0ba4af-fd88-4838-ae7a-2ce433e29b4a","Type":"ContainerStarted","Data":"a592d40173d47bf45a0b07831e72024637971aabd41b87fca7d778a61e8e9e6e"}
Jul 29 10:30:01.055648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.055596    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-49kqc" podStartSLOduration=2.39100377 podStartE2EDuration="8.055575696s" totalImagesPullingTime="5.664571926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.054567323 +0000 UTC m=+52.956795719" watchObservedRunningTime="2026-07-29 10:30:01.055575696 +0000 UTC m=+52.957804138"
Jul 29 10:30:01.072535 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:01.072478    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s5cqz" podStartSLOduration=2.067231372 podStartE2EDuration="8.072461039s" totalImagesPullingTime="6.005229667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:01.071637685 +0000 UTC m=+52.973866088" watchObservedRunningTime="2026-07-29 10:30:01.072461039 +0000 UTC m=+52.974689436"
Jul 29 10:30:02.044824 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.044756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b" event={"ID":"b09bf49f-251c-4e50-9a24-6e711b73e16f","Type":"ContainerStarted","Data":"2e3d8a37c231aef82082f6b4ea3177a0417a02b5ed3d995b392f524918dd010c"}
Jul 29 10:30:02.044824 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.044818    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:30:02.051630 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.051596    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b"
Jul 29 10:30:02.064023 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.063961    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9248b" podStartSLOduration=4.78059097 podStartE2EDuration="6.063935842s" totalImagesPullingTime="1.283344872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:02.063245562 +0000 UTC m=+53.965473954" watchObservedRunningTime="2026-07-29 10:30:02.063935842 +0000 UTC m=+53.966164240"
Jul 29 10:30:02.188537 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.188503    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:02.190600 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.190515    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.195866 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.195813    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 10:30:02.196032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.195813    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 10:30:02.196032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.195814    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 10:30:02.196450 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.196277    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6pjpl\""
Jul 29 10:30:02.196450 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.196359    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 10:30:02.196450 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.196404    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 10:30:02.196750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.196667    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 10:30:02.196750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.196666    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 10:30:02.201673 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.201650    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 10:30:02.204795 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.204726    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:02.348792 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.348753    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.348965 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.348796    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.348965 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.348875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.348965 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.348920    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.349076 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.348995    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.349076 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.349055    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.349076 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.349072    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdqw\" (UniqueName: \"kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450265 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450217    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdqw\" (UniqueName: \"kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450336    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450362    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450393    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.450743 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.450718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.451306 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.451098    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.451434 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.451114    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.451434 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.451218    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.452024 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.452004    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.455054 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.455027    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.455054 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.455046    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.459341 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.459313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdqw\" (UniqueName: \"kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw\") pod \"console-6ffd987569-9d4kq\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") " pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:02.502245 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:02.502205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:04.492681 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:04.492644    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:04.495967 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:04.495935    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb479f0f_5900_4bca_b173_313701a5ba65.slice/crio-aaffa79cf87f70f3b602690fac78144c03dc5b72b80369701931a3d8965c1f41 WatchSource:0}: Error finding container aaffa79cf87f70f3b602690fac78144c03dc5b72b80369701931a3d8965c1f41: Status 404 returned error can't find the container with id aaffa79cf87f70f3b602690fac78144c03dc5b72b80369701931a3d8965c1f41
Jul 29 10:30:05.055518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:05.055478    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccdw4" event={"ID":"8e611c33-c2c7-49f5-965e-116fa6ae10e4","Type":"ContainerStarted","Data":"26503a452a5d7782ac577d440dc6ad78780a5b9eca8f5dbc5e80b7b1b3a361fa"}
Jul 29 10:30:05.056591 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:05.056566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ffd987569-9d4kq" event={"ID":"fb479f0f-5900-4bca-b173-313701a5ba65","Type":"ContainerStarted","Data":"aaffa79cf87f70f3b602690fac78144c03dc5b72b80369701931a3d8965c1f41"}
Jul 29 10:30:08.983053 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:08.982999    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"]
Jul 29 10:30:09.007203 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.007166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"]
Jul 29 10:30:09.007413 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.007227    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-75wlb"]
Jul 29 10:30:09.007413 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.007276    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.009925 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.009893    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 10:30:09.010100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.009902    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:30:09.010921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.010896    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h49t7\""
Jul 29 10:30:09.010921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.010916    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:30:09.011072 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.010964    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:30:09.011231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.011213    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 10:30:09.012274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.012254    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.014670 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.014646    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:30:09.014768 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.014758    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:30:09.014835 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.014820    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:30:09.015206 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.015056    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j9jh4\""
Jul 29 10:30:09.108512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108471    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856cf\" (UniqueName: \"kubernetes.io/projected/f64cd852-e9fa-49be-8bb0-96531811ca1e-kube-api-access-856cf\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.108512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108516    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-accelerators-collector-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108537    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xqw\" (UniqueName: \"kubernetes.io/projected/07e5545f-be3d-4f06-9658-dbfaee2b7493-kube-api-access-k8xqw\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108601    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108644    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f64cd852-e9fa-49be-8bb0-96531811ca1e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.108726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-metrics-client-ca\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108729    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-wtmp\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-root\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108801    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.108899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.108899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.109094 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108915    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-textfile\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.109094 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.108999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-sys\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210299 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-wtmp\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210345    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-root\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210371    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210412    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210449    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210461    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-root\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210474    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-textfile\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-wtmp\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-sys\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210543    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856cf\" (UniqueName: \"kubernetes.io/projected/f64cd852-e9fa-49be-8bb0-96531811ca1e-kube-api-access-856cf\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-accelerators-collector-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xqw\" (UniqueName: \"kubernetes.io/projected/07e5545f-be3d-4f06-9658-dbfaee2b7493-kube-api-access-k8xqw\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210611    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e5545f-be3d-4f06-9658-dbfaee2b7493-sys\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210625    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210652    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f64cd852-e9fa-49be-8bb0-96531811ca1e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.210841 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210792    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-metrics-client-ca\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.211243 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:30:09.210904    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 10:30:09.211243 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.210945    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-textfile\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.211243 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:30:09.210970    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls podName:07e5545f-be3d-4f06-9658-dbfaee2b7493 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:09.710949328 +0000 UTC m=+61.613177721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls") pod "node-exporter-75wlb" (UID: "07e5545f-be3d-4f06-9658-dbfaee2b7493") : secret "node-exporter-tls" not found
Jul 29 10:30:09.211243 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:30:09.211040    2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 10:30:09.211243 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:30:09.211106    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls podName:f64cd852-e9fa-49be-8bb0-96531811ca1e nodeName:}" failed. No retries permitted until 2026-07-29 10:30:09.711087645 +0000 UTC m=+61.613316021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-fz27k" (UID: "f64cd852-e9fa-49be-8bb0-96531811ca1e") : secret "openshift-state-metrics-tls" not found
Jul 29 10:30:09.211448 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.211380    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f64cd852-e9fa-49be-8bb0-96531811ca1e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.211582 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.211558    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-metrics-client-ca\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.211823 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.211801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-accelerators-collector-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.213688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.213659    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.213785 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.213716    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.222605 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.222573    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-856cf\" (UniqueName: \"kubernetes.io/projected/f64cd852-e9fa-49be-8bb0-96531811ca1e-kube-api-access-856cf\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.222605 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.222598    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xqw\" (UniqueName: \"kubernetes.io/projected/07e5545f-be3d-4f06-9658-dbfaee2b7493-kube-api-access-k8xqw\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.715218 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.715181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.715402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.715242    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.717679 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.717657    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07e5545f-be3d-4f06-9658-dbfaee2b7493-node-exporter-tls\") pod \"node-exporter-75wlb\" (UID: \"07e5545f-be3d-4f06-9658-dbfaee2b7493\") " pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.717758 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.717744    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f64cd852-e9fa-49be-8bb0-96531811ca1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-fz27k\" (UID: \"f64cd852-e9fa-49be-8bb0-96531811ca1e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.921167 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.921121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"
Jul 29 10:30:09.928037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.928007    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75wlb"
Jul 29 10:30:09.942423 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:09.942380    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e5545f_be3d_4f06_9658_dbfaee2b7493.slice/crio-c9368df89b5b57ca3555d83a99fb9ecec3da922132e85d27160c77bc341ea445 WatchSource:0}: Error finding container c9368df89b5b57ca3555d83a99fb9ecec3da922132e85d27160c77bc341ea445: Status 404 returned error can't find the container with id c9368df89b5b57ca3555d83a99fb9ecec3da922132e85d27160c77bc341ea445
Jul 29 10:30:09.972732 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:09.972644    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k49lk"
Jul 29 10:30:10.066680 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.066649    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k"]
Jul 29 10:30:10.070606 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:10.070569    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64cd852_e9fa_49be_8bb0_96531811ca1e.slice/crio-0e0c972ea2af9e24150c4c17582422669caafe138bb9c05ae5e4d014e29eeecc WatchSource:0}: Error finding container 0e0c972ea2af9e24150c4c17582422669caafe138bb9c05ae5e4d014e29eeecc: Status 404 returned error can't find the container with id 0e0c972ea2af9e24150c4c17582422669caafe138bb9c05ae5e4d014e29eeecc
Jul 29 10:30:10.071056 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.071040    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:10.076544 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.076522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.076666 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.076609    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccdw4" event={"ID":"8e611c33-c2c7-49f5-965e-116fa6ae10e4","Type":"ContainerStarted","Data":"3102137c25e176ba1333c25677b4e5e8170d03097f6d7edf582604782c45fb5d"}
Jul 29 10:30:10.077981 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.077953    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75wlb" event={"ID":"07e5545f-be3d-4f06-9658-dbfaee2b7493","Type":"ContainerStarted","Data":"c9368df89b5b57ca3555d83a99fb9ecec3da922132e85d27160c77bc341ea445"}
Jul 29 10:30:10.079284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k" event={"ID":"f64cd852-e9fa-49be-8bb0-96531811ca1e","Type":"ContainerStarted","Data":"0e0c972ea2af9e24150c4c17582422669caafe138bb9c05ae5e4d014e29eeecc"}
Jul 29 10:30:10.079392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079308    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t7vqb\""
Jul 29 10:30:10.079578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079558    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 10:30:10.079689 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079645    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 10:30:10.079831 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079807    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 10:30:10.079831 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079818    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 10:30:10.079988 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.079911    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 10:30:10.080099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.080081    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 10:30:10.080187 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.080176    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 10:30:10.080251 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.080203    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 10:30:10.080251 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.080083    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 10:30:10.081178 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.081153    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ffd987569-9d4kq" event={"ID":"fb479f0f-5900-4bca-b173-313701a5ba65","Type":"ContainerStarted","Data":"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"}
Jul 29 10:30:10.093085 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.093059    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:10.120155 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120198    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120241    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-config-volume\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120510 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120572 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120535    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120572 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120563    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120663    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-config-out\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120744 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-web-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120796 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzrq\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-kube-api-access-fqzrq\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120897 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.120955 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.120926    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.121079 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.121048    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.143896 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.143851    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ccdw4" podStartSLOduration=2.444303201 podStartE2EDuration="17.143832853s" totalImagesPullingTime="14.699529652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:10.142823976 +0000 UTC m=+62.045052373" watchObservedRunningTime="2026-07-29 10:30:10.143832853 +0000 UTC m=+62.046061249"
Jul 29 10:30:10.222143 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222151    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-config-volume\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222225    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222265    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222324 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222320    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222360    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-config-out\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-web-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222427    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzrq\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-kube-api-access-fqzrq\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222545    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.222966 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.222642    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.225688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.224745    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.225688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.224715    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8104c6a7-e067-4125-af70-f9373ba7a66b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.226663 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.226632    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.227516 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.227490    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-config-volume\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.227837 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.227794    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.234014 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.231810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.234722 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.234692    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.235350 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.235290    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8104c6a7-e067-4125-af70-f9373ba7a66b-config-out\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.236821 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.236790    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.236922 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.236791    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-web-config\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.236922 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.236791    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8104c6a7-e067-4125-af70-f9373ba7a66b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.237802 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.237779    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzrq\" (UniqueName: \"kubernetes.io/projected/8104c6a7-e067-4125-af70-f9373ba7a66b-kube-api-access-fqzrq\") pod \"alertmanager-main-0\" (UID: \"8104c6a7-e067-4125-af70-f9373ba7a66b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.387628 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.387591    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 10:30:10.546328 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.546204    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ffd987569-9d4kq" podStartSLOduration=3.665191937 podStartE2EDuration="8.546180876s" totalImagesPullingTime="4.880988939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:10.16255885 +0000 UTC m=+62.064787255" watchObservedRunningTime="2026-07-29 10:30:10.546180876 +0000 UTC m=+62.448409274"
Jul 29 10:30:10.546663 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:10.546633    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 10:30:10.551849 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:10.551821    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8104c6a7_e067_4125_af70_f9373ba7a66b.slice/crio-76af8b9427cb7a0976446b063523493ff589cad55540d26ee0d8cc9d920c69e5 WatchSource:0}: Error finding container 76af8b9427cb7a0976446b063523493ff589cad55540d26ee0d8cc9d920c69e5: Status 404 returned error can't find the container with id 76af8b9427cb7a0976446b063523493ff589cad55540d26ee0d8cc9d920c69e5
Jul 29 10:30:11.046888 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:11.046855    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-49kqc"
Jul 29 10:30:11.092083 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:11.092045    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k" event={"ID":"f64cd852-e9fa-49be-8bb0-96531811ca1e","Type":"ContainerStarted","Data":"fe24d4be0ef5f7ab0e403d5fc9e8b49718ebed9510626af951183d357c84e521"}
Jul 29 10:30:11.092594 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:11.092146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k" event={"ID":"f64cd852-e9fa-49be-8bb0-96531811ca1e","Type":"ContainerStarted","Data":"7ce37eab07fad9cdcfb2466286eaa88949d6a6f58ce502df3f05abf5113c1182"}
Jul 29 10:30:11.093811 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:11.093700    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"76af8b9427cb7a0976446b063523493ff589cad55540d26ee0d8cc9d920c69e5"}
Jul 29 10:30:12.100301 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.100258    2551 generic.go:356] "Generic (PLEG): container finished" podID="07e5545f-be3d-4f06-9658-dbfaee2b7493" containerID="dfa3384e9a1cfd4cfca3509c05e78f16cf7656fa44e26ed352ae57c384b1e3d8" exitCode=0
Jul 29 10:30:12.100746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.100349    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75wlb" event={"ID":"07e5545f-be3d-4f06-9658-dbfaee2b7493","Type":"ContainerDied","Data":"dfa3384e9a1cfd4cfca3509c05e78f16cf7656fa44e26ed352ae57c384b1e3d8"}
Jul 29 10:30:12.100746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.100693    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-748f5c9894-8fk2t"]
Jul 29 10:30:12.105269 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.105186    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k" event={"ID":"f64cd852-e9fa-49be-8bb0-96531811ca1e","Type":"ContainerStarted","Data":"1318ec4886416c0ef58927c8d55d2d400158446d0f8b5e1a3bdc01cd3394b07e"}
Jul 29 10:30:12.105269 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.105235    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"c75cf0a126d7741709da14490b49aeb810bd06966027f82bbb5f3c880af08d6e"}
Jul 29 10:30:12.105269 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.105240    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.108400 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.108375    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 10:30:12.108594 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.108581    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 10:30:12.109333 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.109318    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 10:30:12.109441 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.109382    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cxpct\""
Jul 29 10:30:12.109441 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.109382    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 10:30:12.109756 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.109741    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 10:30:12.109863 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.109838    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-49b4okj17llvh\""
Jul 29 10:30:12.123292 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.123259    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-748f5c9894-8fk2t"]
Jul 29 10:30:12.139772 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.139713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.139932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.139796    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140085 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-metrics-client-ca\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140085 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7g7p\" (UniqueName: \"kubernetes.io/projected/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-kube-api-access-w7g7p\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140195 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140122    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-grpc-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140296 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140270    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.140769 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.140738    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.218108 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.218052    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-fz27k" podStartSLOduration=2.9860720819999997 podStartE2EDuration="4.21803209s" totalImagesPullingTime="1.231960008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:12.217549868 +0000 UTC m=+64.119778267" watchObservedRunningTime="2026-07-29 10:30:12.21803209 +0000 UTC m=+64.120260465"
Jul 29 10:30:12.241719 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.241719 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.241918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241755    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.241918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241814    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-metrics-client-ca\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.241918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7g7p\" (UniqueName: \"kubernetes.io/projected/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-kube-api-access-w7g7p\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.242149 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-grpc-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.242149 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.241961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.242258 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.242230    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.242746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.242713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-metrics-client-ca\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.244644 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.244620    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.245062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.245037    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.245182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.245072    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.245253 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.245204    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.245410 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.245382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-thanos-querier-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.245469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.245419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-secret-grpc-tls\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.255676 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.255645    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7g7p\" (UniqueName: \"kubernetes.io/projected/5f6b04c1-17f5-4dd3-93e7-483db67c5fcb-kube-api-access-w7g7p\") pod \"thanos-querier-748f5c9894-8fk2t\" (UID: \"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb\") " pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.462897 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.462857    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:12.503114 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.503070    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:12.503338 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.503185    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:12.508288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.508261    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:12.603149 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:12.603038    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-748f5c9894-8fk2t"]
Jul 29 10:30:12.605732 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:12.605705    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6b04c1_17f5_4dd3_93e7_483db67c5fcb.slice/crio-89874f5da58d6cdfcc539f38ca4c9075c3c518e6b299c3903c724af27f70e32a WatchSource:0}: Error finding container 89874f5da58d6cdfcc539f38ca4c9075c3c518e6b299c3903c724af27f70e32a: Status 404 returned error can't find the container with id 89874f5da58d6cdfcc539f38ca4c9075c3c518e6b299c3903c724af27f70e32a
Jul 29 10:30:13.111074 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.111041    2551 generic.go:356] "Generic (PLEG): container finished" podID="8104c6a7-e067-4125-af70-f9373ba7a66b" containerID="c75cf0a126d7741709da14490b49aeb810bd06966027f82bbb5f3c880af08d6e" exitCode=0
Jul 29 10:30:13.111551 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.111152    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerDied","Data":"c75cf0a126d7741709da14490b49aeb810bd06966027f82bbb5f3c880af08d6e"}
Jul 29 10:30:13.112685 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.112653    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"89874f5da58d6cdfcc539f38ca4c9075c3c518e6b299c3903c724af27f70e32a"}
Jul 29 10:30:13.115323 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.115296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75wlb" event={"ID":"07e5545f-be3d-4f06-9658-dbfaee2b7493","Type":"ContainerStarted","Data":"579cca84d37b99e5a68d36aa66bae36cd9bb4d0af2e79f6c4a3ff8cefe368872"}
Jul 29 10:30:13.115420 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.115350    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75wlb" event={"ID":"07e5545f-be3d-4f06-9658-dbfaee2b7493","Type":"ContainerStarted","Data":"329e0eb71a94c960cec6b6433a616147f5d4ed1807112975a393824573eee029"}
Jul 29 10:30:13.120368 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.120345    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:13.184610 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.184546    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-75wlb" podStartSLOduration=3.615498296 podStartE2EDuration="5.184525598s" totalImagesPullingTime="1.485964657s" totalInitContainerRuntime="83.062645ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:13.184006747 +0000 UTC m=+65.086235175" watchObservedRunningTime="2026-07-29 10:30:13.184525598 +0000 UTC m=+65.086753996"
Jul 29 10:30:13.352190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.352121    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:30:13.354925 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.354898    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:30:13.365969 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.365859    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de60eff5-1df5-4c1b-b19b-3ca6c0f2479a-metrics-certs\") pod \"network-metrics-daemon-9tsjs\" (UID: \"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a\") " pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:30:13.413066 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.413022    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54dcf68874-2bvlv"]
Jul 29 10:30:13.415666 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.415638    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.418378 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418342    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 10:30:13.418521 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418377    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 10:30:13.418521 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418394    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 10:30:13.418750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418718    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 10:30:13.418750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418737    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-47hruam0chlpc\""
Jul 29 10:30:13.418894 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.418788    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rl6kd\""
Jul 29 10:30:13.428040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.428016    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54dcf68874-2bvlv"]
Jul 29 10:30:13.439930 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.439899    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg7mn\""
Jul 29 10:30:13.448281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.448248    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tsjs"
Jul 29 10:30:13.452747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452710    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-client-certs\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.452868 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452763    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:30:13.452868 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452790    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-metrics-server-audit-profiles\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.452868 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452826    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-client-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.452868 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452849    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmnj\" (UniqueName: \"kubernetes.io/projected/7dde4a0c-1abb-417e-87b0-6d74828ac094-kube-api-access-4wmnj\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.453027 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452879    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-tls\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.453027 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452914    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dde4a0c-1abb-417e-87b0-6d74828ac094-audit-log\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.453027 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.452935    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.455578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.455558    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:30:13.466343 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.466317    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/299a9e33-f924-4aea-b3b0-15bde11145dc-original-pull-secret\") pod \"global-pull-secret-syncer-p5v7t\" (UID: \"299a9e33-f924-4aea-b3b0-15bde11145dc\") " pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:30:13.553744 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-client-certs\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-metrics-server-audit-profiles\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553811    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-client-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553839    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmnj\" (UniqueName: \"kubernetes.io/projected/7dde4a0c-1abb-417e-87b0-6d74828ac094-kube-api-access-4wmnj\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-tls\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553915    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553948    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dde4a0c-1abb-417e-87b0-6d74828ac094-audit-log\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.554457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.553978    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.555389 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.554745    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.555630 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.555598    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dde4a0c-1abb-417e-87b0-6d74828ac094-metrics-server-audit-profiles\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.555750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.555731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dde4a0c-1abb-417e-87b0-6d74828ac094-audit-log\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.556855 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.556831    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-client-certs\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.557471 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.556958    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:30:13.557471 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.557427    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-secret-metrics-server-tls\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.557855 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.557830    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dde4a0c-1abb-417e-87b0-6d74828ac094-client-ca-bundle\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.567794 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.567754    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:30:13.569616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.569589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmnj\" (UniqueName: \"kubernetes.io/projected/7dde4a0c-1abb-417e-87b0-6d74828ac094-kube-api-access-4wmnj\") pod \"metrics-server-54dcf68874-2bvlv\" (UID: \"7dde4a0c-1abb-417e-87b0-6d74828ac094\") " pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.578868 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.578833    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnj8m\" (UniqueName: \"kubernetes.io/projected/df2f6da7-779a-4c19-a683-7d97b8d6eac2-kube-api-access-nnj8m\") pod \"network-check-target-89sxt\" (UID: \"df2f6da7-779a-4c19-a683-7d97b8d6eac2\") " pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:30:13.599690 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.599649    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tsjs"]
Jul 29 10:30:13.606159 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:13.606101    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde60eff5_1df5_4c1b_b19b_3ca6c0f2479a.slice/crio-ea0453dd6af86bbd1989599d43eb2c17f957365eeec1f754b632d197f3f03adb WatchSource:0}: Error finding container ea0453dd6af86bbd1989599d43eb2c17f957365eeec1f754b632d197f3f03adb: Status 404 returned error can't find the container with id ea0453dd6af86bbd1989599d43eb2c17f957365eeec1f754b632d197f3f03adb
Jul 29 10:30:13.634294 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.634196    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5v7t"
Jul 29 10:30:13.642037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.642011    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9npp\""
Jul 29 10:30:13.650456 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.650413    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:30:13.727803 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.727766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:13.791624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.791584    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"]
Jul 29 10:30:13.794380 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.794257    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.797473 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.797450    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qsls9\""
Jul 29 10:30:13.797625 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.797555    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 10:30:13.807504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.807478    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"]
Jul 29 10:30:13.857712 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.857598    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42tc\" (UniqueName: \"kubernetes.io/projected/af603e8f-d435-4811-a831-6fd9f980db74-kube-api-access-l42tc\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.857712 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.857689    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af603e8f-d435-4811-a831-6fd9f980db74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.958943 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.958903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42tc\" (UniqueName: \"kubernetes.io/projected/af603e8f-d435-4811-a831-6fd9f980db74-kube-api-access-l42tc\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.959182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.958979    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af603e8f-d435-4811-a831-6fd9f980db74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.962441 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.962413    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af603e8f-d435-4811-a831-6fd9f980db74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:13.968193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:13.968164    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42tc\" (UniqueName: \"kubernetes.io/projected/af603e8f-d435-4811-a831-6fd9f980db74-kube-api-access-l42tc\") pod \"monitoring-plugin-68b7fc9574-6tb7s\" (UID: \"af603e8f-d435-4811-a831-6fd9f980db74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:14.108484 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.108442    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:14.120796 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.120742    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tsjs" event={"ID":"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a","Type":"ContainerStarted","Data":"ea0453dd6af86bbd1989599d43eb2c17f957365eeec1f754b632d197f3f03adb"}
Jul 29 10:30:14.786767 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.786706    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54dcf68874-2bvlv"]
Jul 29 10:30:14.816199 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.815984    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"]
Jul 29 10:30:14.819669 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.819329    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89sxt"]
Jul 29 10:30:14.822494 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:30:14.822458    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2f6da7_779a_4c19_a683_7d97b8d6eac2.slice/crio-24b0abfab8915f9341345a7008409f56837fcf7e282b51104ce868a9858b8e9d WatchSource:0}: Error finding container 24b0abfab8915f9341345a7008409f56837fcf7e282b51104ce868a9858b8e9d: Status 404 returned error can't find the container with id 24b0abfab8915f9341345a7008409f56837fcf7e282b51104ce868a9858b8e9d
Jul 29 10:30:14.860671 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:14.860640    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5v7t"]
Jul 29 10:30:15.129064 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.128972    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"7da6dd56fae520a1b2a0beaa142571098c44b19cb50c113be82cc250e3967663"}
Jul 29 10:30:15.129064 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.129047    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"516d9df7b364165b4b23c29908fc41db3d4c093bb4482203bd2532cb73e0e7c9"}
Jul 29 10:30:15.129529 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.129084    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"58b8e73678d6a7b10ea9a333c9b5aae6add71140eb70f1785e38342aacd61b1c"}
Jul 29 10:30:15.129529 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.129116    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"6e23bec632c534561002df90f8ecc7827f1c9f2c988dbc722f82ec069b842980"}
Jul 29 10:30:15.129529 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.129171    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"37afaa616a36536b2fad6cb9c295840a4fdd3e32d32bfc71687b0f5cffd16590"}
Jul 29 10:30:15.130260 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.130231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s" event={"ID":"af603e8f-d435-4811-a831-6fd9f980db74","Type":"ContainerStarted","Data":"cb5d6d7b68e87da5c17ae03c0bfb56c57b2f1053c17f3f15a405ae8d641ef1ec"}
Jul 29 10:30:15.131532 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.131509    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89sxt" event={"ID":"df2f6da7-779a-4c19-a683-7d97b8d6eac2","Type":"ContainerStarted","Data":"24b0abfab8915f9341345a7008409f56837fcf7e282b51104ce868a9858b8e9d"}
Jul 29 10:30:15.133745 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.133722    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"636f5d424943e922fc383b8bc478533ec17e9cbe543a87a9499c642fd41996bd"}
Jul 29 10:30:15.133838 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.133775    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"11c7dcbd8da70fb2d3fd3f36d1e305119abc9fd5ec5d03513744885425b66a25"}
Jul 29 10:30:15.133838 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.133800    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"55a1979b01eef735cd465403484b4ec0e5cd9d77ae70c2a9b02b35ca9a846aba"}
Jul 29 10:30:15.134811 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.134787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5v7t" event={"ID":"299a9e33-f924-4aea-b3b0-15bde11145dc","Type":"ContainerStarted","Data":"05e26c09d57d04491a5e36b08db35ab337dfce9e288c3875895d8d4c2146deea"}
Jul 29 10:30:15.135862 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.135841    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv" event={"ID":"7dde4a0c-1abb-417e-87b0-6d74828ac094","Type":"ContainerStarted","Data":"a49d4b33468ea270e5ec6cfb3795e10da693aaae19dc34d617e03bb4c1892509"}
Jul 29 10:30:15.218890 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.218845    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:15.222198 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.222170    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.225037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.224960    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 10:30:15.225037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.225008    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 10:30:15.226759 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.226739    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227081    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227180    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227224    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227279    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227334    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227355    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 10:30:15.227447 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227367    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 10:30:15.227862 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.227453    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 10:30:15.228633 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.228437    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dct3kjfimv0kp\""
Jul 29 10:30:15.228633 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.228447    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 10:30:15.228833 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.228676    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zngpg\""
Jul 29 10:30:15.232311 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.232289    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 10:30:15.234330 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.234305    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 10:30:15.250287 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.250252    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:15.274440 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274399    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mq2q\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-kube-api-access-9mq2q\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274468    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274501    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274533    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274592    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274646    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274683    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274709    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274770    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274801    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274870    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274910    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.274985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.274969    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.275214 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.275001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.275214 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.275047    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.275214 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.275071    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.275214 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.275109    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.275214 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.275158    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376205    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376257    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376323    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376420    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376445    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376480    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376513    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376565    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376591    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376636    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376664    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376701    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mq2q\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-kube-api-access-9mq2q\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376752    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.378490 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376787    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376814    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.376888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.378031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.378909    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379533 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.379093    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.379870 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.379713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.381881 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.381849    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.382007 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.381860    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.382606    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.383250    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.387285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4f0c4a1c-c0c1-401a-b311-95174b47250b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.387298    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.390367    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.394017    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.394119    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.394414    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4f0c4a1c-c0c1-401a-b311-95174b47250b-config-out\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.395852    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.396480    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.396997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.397935 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.397839    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-web-config\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.399877 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.399827    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4f0c4a1c-c0c1-401a-b311-95174b47250b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.400466 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.400437    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mq2q\" (UniqueName: \"kubernetes.io/projected/4f0c4a1c-c0c1-401a-b311-95174b47250b-kube-api-access-9mq2q\") pod \"prometheus-k8s-0\" (UID: \"4f0c4a1c-c0c1-401a-b311-95174b47250b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:15.542984 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:15.542942    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:16.000887 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:16.000853    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 10:30:16.155139 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:16.155031    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tsjs" event={"ID":"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a","Type":"ContainerStarted","Data":"7c781316756d462351e04d76c3a4bfed2ae44914dedd5bb2d7a52d6f936b1ca0"}
Jul 29 10:30:16.159304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:16.159266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"41c91133b4204df791cc2e81c3a0a81500c48edb73065a043758c33bc0bbb791"}
Jul 29 10:30:17.167513 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:17.167349    2551 generic.go:356] "Generic (PLEG): container finished" podID="4f0c4a1c-c0c1-401a-b311-95174b47250b" containerID="5a5a05f2b129eb1383a327f3e6d2ad1e546c99fa910ff82a964aaa5d23ced6a4" exitCode=0
Jul 29 10:30:17.167513 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:17.167437    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerDied","Data":"5a5a05f2b129eb1383a327f3e6d2ad1e546c99fa910ff82a964aaa5d23ced6a4"}
Jul 29 10:30:17.169963 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:17.169924    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tsjs" event={"ID":"de60eff5-1df5-4c1b-b19b-3ca6c0f2479a","Type":"ContainerStarted","Data":"55617b7529907cdb8b073a00f1750b3e15b3f9e4077e03d24d4cddd70194cd12"}
Jul 29 10:30:17.218796 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:17.218530    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9tsjs" podStartSLOduration=65.980629638 podStartE2EDuration="1m8.218505506s" totalImagesPullingTime="2.237875868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:17.218325169 +0000 UTC m=+69.120553567" watchObservedRunningTime="2026-07-29 10:30:17.218505506 +0000 UTC m=+69.120733904"
Jul 29 10:30:22.196467 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.196424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8104c6a7-e067-4125-af70-f9373ba7a66b","Type":"ContainerStarted","Data":"677084be844d64c729c842ba76f46bccbbe30a3829daef815b3a2635e5c34d51"}
Jul 29 10:30:22.198151 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.198100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s" event={"ID":"af603e8f-d435-4811-a831-6fd9f980db74","Type":"ContainerStarted","Data":"63e44d601f6baea26fff169e86cb9b9f0ad512e9b117bafae91aa8be2870eba6"}
Jul 29 10:30:22.198303 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.198182    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:22.199972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.199738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89sxt" event={"ID":"df2f6da7-779a-4c19-a683-7d97b8d6eac2","Type":"ContainerStarted","Data":"b5ae7c5dbccdfeb194e4e4c46b5b77c2a8442fb7350169bde896d07ba76a9c2a"}
Jul 29 10:30:22.200107 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.200005    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:30:22.203952 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.203920    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"928255da35021aa8dcf90748e9c39a767c78fc7236dc9548c929a23cb3049874"}
Jul 29 10:30:22.204108 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"9bd752ed4b695804e6a22d0dbaf54028488739ba6fd9ecd794e5c31c45c660fe"}
Jul 29 10:30:22.204108 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204090    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s"
Jul 29 10:30:22.204245 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204113    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"86aafd43d90447e4e94a885cd266cc60821557b5f06d336577b7727a4fadff71"}
Jul 29 10:30:22.204245 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204182    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"5935606ed4c316749e24325fbd9bcafbb61f0b2ef776698cd1233b9233c30c99"}
Jul 29 10:30:22.204245 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"8b21b70ab609dd36d01ae0800f35f5b21964f5659d8480fc6af0d7d5510c0302"}
Jul 29 10:30:22.204396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.204280    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4f0c4a1c-c0c1-401a-b311-95174b47250b","Type":"ContainerStarted","Data":"06a47fec27d154f44d4ffbd2793c15316407b8555a382e111eab6c8d4e919543"}
Jul 29 10:30:22.206967 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.206942    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"4a0542588dacdfaf6fe0f06913b97aa815e4920f7ff2dc5c6c3ab2c41dfb506f"}
Jul 29 10:30:22.207104 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.207014    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"ee67e8a3be970c1765840e0519efa0ffa66a8cd78523e9e303d0cf43f88fbd75"}
Jul 29 10:30:22.207104 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.207064    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:22.207234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.207108    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" event={"ID":"5f6b04c1-17f5-4dd3-93e7-483db67c5fcb","Type":"ContainerStarted","Data":"85e7e788045b1bb0598afe3dd38336ed89af1c96e1155429182ba8e137f6e30b"}
Jul 29 10:30:22.208630 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.208600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5v7t" event={"ID":"299a9e33-f924-4aea-b3b0-15bde11145dc","Type":"ContainerStarted","Data":"7f289894c7bbd014de9bce3ddc53d291a91d23ff91e34f85fcf7f8f793e78fce"}
Jul 29 10:30:22.210271 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.210240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv" event={"ID":"7dde4a0c-1abb-417e-87b0-6d74828ac094","Type":"ContainerStarted","Data":"856f75b43e200ba7b497bf00f602cfa3d8c1ef3d97dcb5f5320a5627e65312e7"}
Jul 29 10:30:22.213103 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.213085    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t"
Jul 29 10:30:22.233921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.233866    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.914151677 podStartE2EDuration="12.233848122s" totalImagesPullingTime="8.302123288s" totalInitContainerRuntime="17.573157ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.231751923 +0000 UTC m=+74.133980321" watchObservedRunningTime="2026-07-29 10:30:22.233848122 +0000 UTC m=+74.136076519"
Jul 29 10:30:22.251383 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.251325    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p5v7t" podStartSLOduration=66.577627877 podStartE2EDuration="1m13.251304717s" totalImagesPullingTime="6.67367684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.250002101 +0000 UTC m=+74.152230498" watchObservedRunningTime="2026-07-29 10:30:22.251304717 +0000 UTC m=+74.153533115"
Jul 29 10:30:22.269846 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.269720    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv" podStartSLOduration=3.643979457 podStartE2EDuration="9.269692578s" totalImagesPullingTime="5.625713121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.268021869 +0000 UTC m=+74.170250277" watchObservedRunningTime="2026-07-29 10:30:22.269692578 +0000 UTC m=+74.171920977"
Jul 29 10:30:22.298523 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.298464    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.925184556 podStartE2EDuration="7.298445393s" totalImagesPullingTime="4.348819129s" totalInitContainerRuntime="24.441708ms" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.296620393 +0000 UTC m=+74.198848816" watchObservedRunningTime="2026-07-29 10:30:22.298445393 +0000 UTC m=+74.200673769"
Jul 29 10:30:22.314439 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.314369    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-89sxt" podStartSLOduration=66.623380299 podStartE2EDuration="1m13.314349488s" totalImagesPullingTime="6.690969189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.313444868 +0000 UTC m=+74.215673277" watchObservedRunningTime="2026-07-29 10:30:22.314349488 +0000 UTC m=+74.216577885"
Jul 29 10:30:22.338932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.338875    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-748f5c9894-8fk2t" podStartSLOduration=2.847028042 podStartE2EDuration="10.33885396s" totalImagesPullingTime="7.491825918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.337061948 +0000 UTC m=+74.239290387" watchObservedRunningTime="2026-07-29 10:30:22.33885396 +0000 UTC m=+74.241082356"
Jul 29 10:30:22.357121 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:22.357063    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tb7s" podStartSLOduration=3.755915215 podStartE2EDuration="9.357041671s" totalImagesPullingTime="5.601126456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.355371744 +0000 UTC m=+74.257600141" watchObservedRunningTime="2026-07-29 10:30:22.357041671 +0000 UTC m=+74.259270101"
Jul 29 10:30:25.543430 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:25.543390    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:30:25.989432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:25.989394    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:33.728872 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:33.728740    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:33.728872 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:33.728818    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:51.008890 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.008828    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ffd987569-9d4kq" podUID="fb479f0f-5900-4bca-b173-313701a5ba65" containerName="console" containerID="cri-o://e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb" gracePeriod=15
Jul 29 10:30:51.244366 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.244344    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:51.296769 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296700    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296769 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296761    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmdqw\" (UniqueName: \"kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296808    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296850    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296873    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296900    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.296948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.296923    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle\") pod \"fb479f0f-5900-4bca-b173-313701a5ba65\" (UID: \"fb479f0f-5900-4bca-b173-313701a5ba65\") "
Jul 29 10:30:51.297212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.297185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:51.297275 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.297251    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config" (OuterVolumeSpecName: "console-config") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:51.297332 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.297249    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:51.297544 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.297520    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:51.299171 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.299139    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw" (OuterVolumeSpecName: "kube-api-access-lmdqw") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "kube-api-access-lmdqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:51.299268 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.299159    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:51.299268 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.299202    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb479f0f-5900-4bca-b173-313701a5ba65" (UID: "fb479f0f-5900-4bca-b173-313701a5ba65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:51.328614 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.328588    2551 generic.go:356] "Generic (PLEG): container finished" podID="fb479f0f-5900-4bca-b173-313701a5ba65" containerID="e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb" exitCode=0
Jul 29 10:30:51.328716 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.328626    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ffd987569-9d4kq" event={"ID":"fb479f0f-5900-4bca-b173-313701a5ba65","Type":"ContainerDied","Data":"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"}
Jul 29 10:30:51.328716 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.328631    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ffd987569-9d4kq"
Jul 29 10:30:51.328716 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.328661    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ffd987569-9d4kq" event={"ID":"fb479f0f-5900-4bca-b173-313701a5ba65","Type":"ContainerDied","Data":"aaffa79cf87f70f3b602690fac78144c03dc5b72b80369701931a3d8965c1f41"}
Jul 29 10:30:51.328716 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.328680    2551 scope.go:117] "RemoveContainer" containerID="e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"
Jul 29 10:30:51.338919 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.338903    2551 scope.go:117] "RemoveContainer" containerID="e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"
Jul 29 10:30:51.339203 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:30:51.339174    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb\": container with ID starting with e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb not found: ID does not exist" containerID="e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"
Jul 29 10:30:51.339296 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.339209    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb"} err="failed to get container status \"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb\": rpc error: code = NotFound desc = could not find container \"e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb\": container with ID starting with e9e3bdc8d933dd2b666542706685d357c76732eb98a73cfdde75e1b032a77ffb not found: ID does not exist"
Jul 29 10:30:51.354249 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.354229    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:51.358674 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.358655    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ffd987569-9d4kq"]
Jul 29 10:30:51.397834 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397815    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdqw\" (UniqueName: \"kubernetes.io/projected/fb479f0f-5900-4bca-b173-313701a5ba65-kube-api-access-lmdqw\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397834 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397833    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-serving-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397844    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-console-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397853    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-service-ca\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397860    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb479f0f-5900-4bca-b173-313701a5ba65-console-oauth-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397868    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-trusted-ca-bundle\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:51.397932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:51.397876    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb479f0f-5900-4bca-b173-313701a5ba65-oauth-serving-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:30:52.727667 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:52.727634    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb479f0f-5900-4bca-b173-313701a5ba65" path="/var/lib/kubelet/pods/fb479f0f-5900-4bca-b173-313701a5ba65/volumes"
Jul 29 10:30:53.217861 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:53.217834    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89sxt"
Jul 29 10:30:53.733749 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:53.733721    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:30:53.737488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:30:53.737468    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54dcf68874-2bvlv"
Jul 29 10:31:15.543824 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:31:15.543787    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:15.565885 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:31:15.565859    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:31:16.442937 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:31:16.442912    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 10:34:08.530453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:34:08.530416    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:34:08.534233 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:34:08.534208    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:34:08.536331 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:34:08.536304    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:36:20.286007 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.285970    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-4fv2c"]
Jul 29 10:36:20.286525 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.286421    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb479f0f-5900-4bca-b173-313701a5ba65" containerName="console"
Jul 29 10:36:20.286525 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.286440    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb479f0f-5900-4bca-b173-313701a5ba65" containerName="console"
Jul 29 10:36:20.286643 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.286528    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb479f0f-5900-4bca-b173-313701a5ba65" containerName="console"
Jul 29 10:36:20.288314 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.288295    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.291305 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.291284    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 10:36:20.292317 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.292299    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fp5lb\""
Jul 29 10:36:20.292392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.292299    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 10:36:20.292392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.292305    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 10:36:20.297615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.297597    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4fv2c"]
Jul 29 10:36:20.391032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.391006    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgfp\" (UniqueName: \"kubernetes.io/projected/a94dfa90-f765-4ffa-90c4-330a38400d94-kube-api-access-gpgfp\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.391166 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.391036    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a94dfa90-f765-4ffa-90c4-330a38400d94-data\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.492035 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.492006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgfp\" (UniqueName: \"kubernetes.io/projected/a94dfa90-f765-4ffa-90c4-330a38400d94-kube-api-access-gpgfp\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.492183 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.492040    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a94dfa90-f765-4ffa-90c4-330a38400d94-data\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.492426 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.492404    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a94dfa90-f765-4ffa-90c4-330a38400d94-data\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.500498 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.500479    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgfp\" (UniqueName: \"kubernetes.io/projected/a94dfa90-f765-4ffa-90c4-330a38400d94-kube-api-access-gpgfp\") pod \"seaweedfs-64568bcd49-4fv2c\" (UID: \"a94dfa90-f765-4ffa-90c4-330a38400d94\") " pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.597406 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.597350    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:20.728144 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:36:20.728096    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94dfa90_f765_4ffa_90c4_330a38400d94.slice/crio-3c3228acc9bcfd00307f777faf50d9899bf5c8fe33f460beaea18d5f13fc5586 WatchSource:0}: Error finding container 3c3228acc9bcfd00307f777faf50d9899bf5c8fe33f460beaea18d5f13fc5586: Status 404 returned error can't find the container with id 3c3228acc9bcfd00307f777faf50d9899bf5c8fe33f460beaea18d5f13fc5586
Jul 29 10:36:20.728391 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.728371    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-4fv2c"]
Jul 29 10:36:20.729328 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:20.729311    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:36:21.556948 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:21.556915    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4fv2c" event={"ID":"a94dfa90-f765-4ffa-90c4-330a38400d94","Type":"ContainerStarted","Data":"3c3228acc9bcfd00307f777faf50d9899bf5c8fe33f460beaea18d5f13fc5586"}
Jul 29 10:36:24.570318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:24.570283    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-4fv2c" event={"ID":"a94dfa90-f765-4ffa-90c4-330a38400d94","Type":"ContainerStarted","Data":"bfe8c0cb39b9f675fdef17d5011269bcc836c5547dc50aba0d275423225cd564"}
Jul 29 10:36:24.570698 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:24.570508    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:36:24.594392 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:24.594350    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-4fv2c" podStartSLOduration=1.785884831 podStartE2EDuration="4.594335608s" totalImagesPullingTime="2.808450777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:36:24.593774386 +0000 UTC m=+436.496002782" watchObservedRunningTime="2026-07-29 10:36:24.594335608 +0000 UTC m=+436.496564005"
Jul 29 10:36:30.576713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:36:30.576636    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-4fv2c"
Jul 29 10:37:36.408860 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.408831    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-n9nqr"]
Jul 29 10:37:36.411822 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.411806    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.414100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.414078    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 29 10:37:36.414274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.414123    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5c6qf\""
Jul 29 10:37:36.421955 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.421933    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n9nqr"]
Jul 29 10:37:36.522308 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.522284    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wtqr\" (UniqueName: \"kubernetes.io/projected/97115e08-1ec0-438d-8d84-15792117f185-kube-api-access-9wtqr\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.522404 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.522377    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97115e08-1ec0-438d-8d84-15792117f185-tls-certs\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.623548 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.623522    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97115e08-1ec0-438d-8d84-15792117f185-tls-certs\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.623646 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.623572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wtqr\" (UniqueName: \"kubernetes.io/projected/97115e08-1ec0-438d-8d84-15792117f185-kube-api-access-9wtqr\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.625696 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.625675    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97115e08-1ec0-438d-8d84-15792117f185-tls-certs\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.630737 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.630702    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wtqr\" (UniqueName: \"kubernetes.io/projected/97115e08-1ec0-438d-8d84-15792117f185-kube-api-access-9wtqr\") pod \"model-serving-api-7b954ffc95-n9nqr\" (UID: \"97115e08-1ec0-438d-8d84-15792117f185\") " pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.722590 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.722568    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:36.849770 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:36.849662    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n9nqr"]
Jul 29 10:37:36.852496 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:37:36.852471    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97115e08_1ec0_438d_8d84_15792117f185.slice/crio-c345f4bb23b2076868e3e7caf7f593c9a35d16fbac079aaf455749357fd5f407 WatchSource:0}: Error finding container c345f4bb23b2076868e3e7caf7f593c9a35d16fbac079aaf455749357fd5f407: Status 404 returned error can't find the container with id c345f4bb23b2076868e3e7caf7f593c9a35d16fbac079aaf455749357fd5f407
Jul 29 10:37:37.849573 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:37.849536    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n9nqr" event={"ID":"97115e08-1ec0-438d-8d84-15792117f185","Type":"ContainerStarted","Data":"c345f4bb23b2076868e3e7caf7f593c9a35d16fbac079aaf455749357fd5f407"}
Jul 29 10:37:39.859554 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:39.859525    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n9nqr" event={"ID":"97115e08-1ec0-438d-8d84-15792117f185","Type":"ContainerStarted","Data":"e5828c59f7dbef996f04b1641b69ab64b710663a6bc27ccf24d12e3f776adadc"}
Jul 29 10:37:39.859943 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:39.859661    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:39.877877 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:39.877838    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-n9nqr" podStartSLOduration=1.079833706 podStartE2EDuration="3.877825344s" totalImagesPullingTime="2.797991638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:39.87766382 +0000 UTC m=+511.779892218" watchObservedRunningTime="2026-07-29 10:37:39.877825344 +0000 UTC m=+511.780053740"
Jul 29 10:37:50.866586 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:50.866555    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-n9nqr"
Jul 29 10:37:52.368854 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.368814    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vvhm9"]
Jul 29 10:37:52.372307 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.372285    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.379645 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.379616    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vvhm9"]
Jul 29 10:37:52.443976 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.443949    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzsq\" (UniqueName: \"kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.444086 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.444012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.544935 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.544909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.545032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.544947    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzsq\" (UniqueName: \"kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.545274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.545257    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.552794 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.552774    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzsq\" (UniqueName: \"kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq\") pod \"s3-init-vvhm9\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") " pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.691918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.691863    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vvhm9"
Jul 29 10:37:52.818997 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.818829    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vvhm9"]
Jul 29 10:37:52.821588 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:37:52.821561    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e5234fa_dc2c_44ed_aa70_2a4ab10a9d4c.slice/crio-558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163 WatchSource:0}: Error finding container 558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163: Status 404 returned error can't find the container with id 558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163
Jul 29 10:37:52.910346 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:52.910321    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerStarted","Data":"558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163"}
Jul 29 10:37:56.929693 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:56.929658    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerStarted","Data":"2334d7d22c12ed9e591b143f3a555e7e9b39c445c374cfb7b42d85a8df2908ff"}
Jul 29 10:37:57.934664 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:57.934632    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerID="2334d7d22c12ed9e591b143f3a555e7e9b39c445c374cfb7b42d85a8df2908ff" exitCode=0
Jul 29 10:37:57.934973 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:37:57.934717    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerDied","Data":"2334d7d22c12ed9e591b143f3a555e7e9b39c445c374cfb7b42d85a8df2908ff"}
Jul 29 10:38:02.963123 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:38:02.963090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerStarted","Data":"07b5ac8ca145e7c7485cfff1549c01004bcb9c33149b58405c8288ca38010b7c"}
Jul 29 10:38:02.980756 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:38:02.980708    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vvhm9" podStartSLOduration=1.378641849 podStartE2EDuration="10.980692144s" totalImagesPullingTime="7.97082959s" totalInitContainerRuntime="1.631220705s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:02.980038892 +0000 UTC m=+534.882267289" watchObservedRunningTime="2026-07-29 10:38:02.980692144 +0000 UTC m=+534.882920541"
Jul 29 10:39:08.224487 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:08.224452    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerID="07b5ac8ca145e7c7485cfff1549c01004bcb9c33149b58405c8288ca38010b7c" exitCode=0
Jul 29 10:39:08.224864 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:08.224521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerDied","Data":"07b5ac8ca145e7c7485cfff1549c01004bcb9c33149b58405c8288ca38010b7c"}
Jul 29 10:39:08.566977 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:08.566905    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:39:08.567661 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:08.567642    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:39:09.352847 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.352826    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vvhm9"
Jul 29 10:39:09.428729 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.428707    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache\") pod \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") "
Jul 29 10:39:09.428850 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.428740    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rzsq\" (UniqueName: \"kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq\") pod \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\" (UID: \"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c\") "
Jul 29 10:39:09.430712 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.430691    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq" (OuterVolumeSpecName: "kube-api-access-5rzsq") pod "2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" (UID: "2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c"). InnerVolumeSpecName "kube-api-access-5rzsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:39:09.447500 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.447477    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" (UID: "2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:39:09.530110 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.530063    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-model-cache\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:39:09.530110 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:09.530084    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzsq\" (UniqueName: \"kubernetes.io/projected/2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c-kube-api-access-5rzsq\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:39:10.235193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:10.235168    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vvhm9"
Jul 29 10:39:10.235383 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:10.235166    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vvhm9" event={"ID":"2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c","Type":"ContainerDied","Data":"558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163"}
Jul 29 10:39:10.235383 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:10.235275    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="558b4731883f99666d5ae195ebb8f9fd570a7f358463b27f30c77e1a42d77163"
Jul 29 10:39:45.119738 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.119664    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"]
Jul 29 10:39:45.120212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.119963    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerName="download-hf-model"
Jul 29 10:39:45.120212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.119975    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerName="download-hf-model"
Jul 29 10:39:45.120212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.119985    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerName="s3-init"
Jul 29 10:39:45.120212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.119992    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerName="s3-init"
Jul 29 10:39:45.120212 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.120053    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c" containerName="s3-init"
Jul 29 10:39:45.122925 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.122910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.125539 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.125519    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 29 10:39:45.125676 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.125577    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 29 10:39:45.131086 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.131066    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"]
Jul 29 10:39:45.215836 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.215806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9487-7446-4b44-b49d-b7d51e045db5-data\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.215990 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.215851    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.215990 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.215873    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8x4\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-kube-api-access-9f8x4\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.316580 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.316555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9487-7446-4b44-b49d-b7d51e045db5-data\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.316665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.316590    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.316665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.316609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8x4\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-kube-api-access-9f8x4\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.316923 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.316906    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/11ef9487-7446-4b44-b49d-b7d51e045db5-data\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.319059 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.319025    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.325311 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.325290    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8x4\" (UniqueName: \"kubernetes.io/projected/11ef9487-7446-4b44-b49d-b7d51e045db5-kube-api-access-9f8x4\") pod \"seaweedfs-tls-custom-8b44d767d-j5mmj\" (UID: \"11ef9487-7446-4b44-b49d-b7d51e045db5\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.432244 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.432190    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"
Jul 29 10:39:45.554590 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:45.554556    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-j5mmj"]
Jul 29 10:39:45.557760 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:39:45.557729    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ef9487_7446_4b44_b49d_b7d51e045db5.slice/crio-2a2748d0d470a99cb0ae57a6cc5c4da602fbd30163b6c7eb4e7c10e04f15e15c WatchSource:0}: Error finding container 2a2748d0d470a99cb0ae57a6cc5c4da602fbd30163b6c7eb4e7c10e04f15e15c: Status 404 returned error can't find the container with id 2a2748d0d470a99cb0ae57a6cc5c4da602fbd30163b6c7eb4e7c10e04f15e15c
Jul 29 10:39:46.379046 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.379009    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj" event={"ID":"11ef9487-7446-4b44-b49d-b7d51e045db5","Type":"ContainerStarted","Data":"80e5a95169bc9a8b3d17738e7353f79d020c442e0dc0ccd25f4765a591554ca0"}
Jul 29 10:39:46.379433 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.379067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj" event={"ID":"11ef9487-7446-4b44-b49d-b7d51e045db5","Type":"ContainerStarted","Data":"2a2748d0d470a99cb0ae57a6cc5c4da602fbd30163b6c7eb4e7c10e04f15e15c"}
Jul 29 10:39:46.395004 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.394944    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-j5mmj" podStartSLOduration=1.086807001 podStartE2EDuration="1.394927396s" totalImagesPullingTime="308.120395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:46.394356025 +0000 UTC m=+638.296584422" watchObservedRunningTime="2026-07-29 10:39:46.394927396 +0000 UTC m=+638.297155814"
Jul 29 10:39:46.706529 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.706497    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-rnwlj"]
Jul 29 10:39:46.710820 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.710806    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.717901 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.717823    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rnwlj"]
Jul 29 10:39:46.828409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.828381    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cd8f\" (UniqueName: \"kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.828527 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.828430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.929624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.929597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cd8f\" (UniqueName: \"kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.929734 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.929663    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.929983 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.929964    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:46.938258 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:46.938238    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cd8f\" (UniqueName: \"kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f\") pod \"s3-tls-init-custom-rnwlj\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") " pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:47.029194 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:47.029121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:47.152485 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:47.152463    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rnwlj"]
Jul 29 10:39:47.154922 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:39:47.154898    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7607eeee_1e7e_461f_9580_0a2f65efa14e.slice/crio-0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf WatchSource:0}: Error finding container 0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf: Status 404 returned error can't find the container with id 0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf
Jul 29 10:39:47.386470 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:47.386383    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerStarted","Data":"0be69a150efe40f8a333a09689ae38e4b9480faaed2a4d00ecdcc1045210c96d"}
Jul 29 10:39:47.386470 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:47.386424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerStarted","Data":"0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf"}
Jul 29 10:39:49.398424 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:49.398353    2551 generic.go:356] "Generic (PLEG): container finished" podID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerID="0be69a150efe40f8a333a09689ae38e4b9480faaed2a4d00ecdcc1045210c96d" exitCode=0
Jul 29 10:39:49.398424 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:49.398418    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerDied","Data":"0be69a150efe40f8a333a09689ae38e4b9480faaed2a4d00ecdcc1045210c96d"}
Jul 29 10:39:50.404211 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:50.404175    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerStarted","Data":"29ce5c16c949211db72d0dd922cfba06fe0d10c7b3ef053ee8fa7751039e9a60"}
Jul 29 10:39:50.422516 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:50.422474    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-rnwlj" podStartSLOduration=2.585730145 podStartE2EDuration="4.422458857s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.836728712s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:50.422198245 +0000 UTC m=+642.324426642" watchObservedRunningTime="2026-07-29 10:39:50.422458857 +0000 UTC m=+642.324687253"
Jul 29 10:39:55.430244 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:55.430212    2551 generic.go:356] "Generic (PLEG): container finished" podID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerID="29ce5c16c949211db72d0dd922cfba06fe0d10c7b3ef053ee8fa7751039e9a60" exitCode=0
Jul 29 10:39:55.430542 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:55.430285    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerDied","Data":"29ce5c16c949211db72d0dd922cfba06fe0d10c7b3ef053ee8fa7751039e9a60"}
Jul 29 10:39:56.599200 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.599176    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:56.711227 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.711204    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cd8f\" (UniqueName: \"kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f\") pod \"7607eeee-1e7e-461f-9580-0a2f65efa14e\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") "
Jul 29 10:39:56.711326 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.711272    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache\") pod \"7607eeee-1e7e-461f-9580-0a2f65efa14e\" (UID: \"7607eeee-1e7e-461f-9580-0a2f65efa14e\") "
Jul 29 10:39:56.713139 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.713099    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f" (OuterVolumeSpecName: "kube-api-access-4cd8f") pod "7607eeee-1e7e-461f-9580-0a2f65efa14e" (UID: "7607eeee-1e7e-461f-9580-0a2f65efa14e"). InnerVolumeSpecName "kube-api-access-4cd8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:39:56.733874 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.733851    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7607eeee-1e7e-461f-9580-0a2f65efa14e" (UID: "7607eeee-1e7e-461f-9580-0a2f65efa14e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:39:56.812267 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.812243    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cd8f\" (UniqueName: \"kubernetes.io/projected/7607eeee-1e7e-461f-9580-0a2f65efa14e-kube-api-access-4cd8f\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:39:56.812267 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:56.812267    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7607eeee-1e7e-461f-9580-0a2f65efa14e-model-cache\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:39:57.442573 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:57.442543    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rnwlj"
Jul 29 10:39:57.442573 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:57.442569    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rnwlj" event={"ID":"7607eeee-1e7e-461f-9580-0a2f65efa14e","Type":"ContainerDied","Data":"0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf"}
Jul 29 10:39:57.442764 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:57.442605    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0862d9fb9e97b8ae1c515854ad094b76b160aa6e15a65bff41b2821c5cd126cf"
Jul 29 10:39:58.030199 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030169    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"]
Jul 29 10:39:58.030564 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030464    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerName="download-hf-model"
Jul 29 10:39:58.030564 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030475    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerName="download-hf-model"
Jul 29 10:39:58.030564 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030485    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerName="s3-tls-init-custom"
Jul 29 10:39:58.030564 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030490    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerName="s3-tls-init-custom"
Jul 29 10:39:58.030564 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.030551    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7607eeee-1e7e-461f-9580-0a2f65efa14e" containerName="s3-tls-init-custom"
Jul 29 10:39:58.034391 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.034375    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.036882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.036859    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 29 10:39:58.036938 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.036914    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 29 10:39:58.044341 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.044319    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"]
Jul 29 10:39:58.120613 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.120588    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.120707 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.120623    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e94fa99-8212-4ece-b8c8-5638fdc047af-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.120707 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.120659    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqbw\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-kube-api-access-ndqbw\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.221302 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.221272    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e94fa99-8212-4ece-b8c8-5638fdc047af-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.221380 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.221317    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqbw\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-kube-api-access-ndqbw\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.221429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.221383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.221512 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:39:58.221493    2551 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 29 10:39:58.221546 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:39:58.221517    2551 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k: secret "seaweedfs-tls-serving" not found
Jul 29 10:39:58.221606 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:39:58.221594    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving podName:3e94fa99-8212-4ece-b8c8-5638fdc047af nodeName:}" failed. No retries permitted until 2026-07-29 10:39:58.721572122 +0000 UTC m=+650.623800501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-rwx5k" (UID: "3e94fa99-8212-4ece-b8c8-5638fdc047af") : secret "seaweedfs-tls-serving" not found
Jul 29 10:39:58.221701 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.221684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e94fa99-8212-4ece-b8c8-5638fdc047af-data\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.230005 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.229980    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqbw\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-kube-api-access-ndqbw\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.723898 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.723861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.726325 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.726302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/3e94fa99-8212-4ece-b8c8-5638fdc047af-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-rwx5k\" (UID: \"3e94fa99-8212-4ece-b8c8-5638fdc047af\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:58.943197 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:58.943091    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"
Jul 29 10:39:59.069457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:59.069432    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k"]
Jul 29 10:39:59.071724 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:39:59.071685    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e94fa99_8212_4ece_b8c8_5638fdc047af.slice/crio-5a9eda3d65c28753a270012bbd587b99385e3424d16e293db143f8743b4e565e WatchSource:0}: Error finding container 5a9eda3d65c28753a270012bbd587b99385e3424d16e293db143f8743b4e565e: Status 404 returned error can't find the container with id 5a9eda3d65c28753a270012bbd587b99385e3424d16e293db143f8743b4e565e
Jul 29 10:39:59.451672 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:59.451637    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k" event={"ID":"3e94fa99-8212-4ece-b8c8-5638fdc047af","Type":"ContainerStarted","Data":"44b7559d1fe9077c4f4523977e44f69dbc05ba5d71f7b5bcc17fcf254cb04784"}
Jul 29 10:39:59.451793 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:59.451688    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k" event={"ID":"3e94fa99-8212-4ece-b8c8-5638fdc047af","Type":"ContainerStarted","Data":"5a9eda3d65c28753a270012bbd587b99385e3424d16e293db143f8743b4e565e"}
Jul 29 10:39:59.467750 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:39:59.467702    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-rwx5k" podStartSLOduration=1.2278911319999999 podStartE2EDuration="1.467685036s" totalImagesPullingTime="239.793904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:39:59.466208425 +0000 UTC m=+651.368436820" watchObservedRunningTime="2026-07-29 10:39:59.467685036 +0000 UTC m=+651.369913433"
Jul 29 10:40:00.013236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.013199    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-9pt74"]
Jul 29 10:40:00.016514 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.016491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.025566 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.025537    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9pt74"]
Jul 29 10:40:00.134173 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.134119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.134495 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.134195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scn9v\" (UniqueName: \"kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.235377 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.235356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.235496 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.235394    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scn9v\" (UniqueName: \"kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.235763 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.235742    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.244880 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.244853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scn9v\" (UniqueName: \"kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v\") pod \"s3-tls-init-serving-9pt74\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") " pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.336306 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.336249    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:00.470546 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:00.470519    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9pt74"]
Jul 29 10:40:00.473420 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:40:00.473396    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6159ef75_c8ac_4cc6_ac35_880ac8279972.slice/crio-aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85 WatchSource:0}: Error finding container aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85: Status 404 returned error can't find the container with id aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85
Jul 29 10:40:01.462412 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:01.462377    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerStarted","Data":"a2b225fecf7d8968b5b791a01996e67e4aa3cd104b56559877f01887449031f2"}
Jul 29 10:40:01.462949 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:01.462424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerStarted","Data":"aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85"}
Jul 29 10:40:02.467654 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:02.467626    2551 generic.go:356] "Generic (PLEG): container finished" podID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerID="a2b225fecf7d8968b5b791a01996e67e4aa3cd104b56559877f01887449031f2" exitCode=0
Jul 29 10:40:02.468033 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:02.467715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerDied","Data":"a2b225fecf7d8968b5b791a01996e67e4aa3cd104b56559877f01887449031f2"}
Jul 29 10:40:03.473471 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:03.473438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerStarted","Data":"6f0b8e708de5de6866c48cfd6eff2dcf0b535658d17ad894ba218bac588d5bed"}
Jul 29 10:40:03.489379 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:03.489335    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-9pt74" podStartSLOduration=2.906882725 podStartE2EDuration="4.48931908s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.582436355s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:40:03.489023433 +0000 UTC m=+655.391251825" watchObservedRunningTime="2026-07-29 10:40:03.48931908 +0000 UTC m=+655.391547476"
Jul 29 10:40:09.499678 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:09.499638    2551 generic.go:356] "Generic (PLEG): container finished" podID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerID="6f0b8e708de5de6866c48cfd6eff2dcf0b535658d17ad894ba218bac588d5bed" exitCode=0
Jul 29 10:40:09.500092 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:09.499713    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerDied","Data":"6f0b8e708de5de6866c48cfd6eff2dcf0b535658d17ad894ba218bac588d5bed"}
Jul 29 10:40:10.650090 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.650062    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:10.720290 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.720269    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache\") pod \"6159ef75-c8ac-4cc6-ac35-880ac8279972\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") "
Jul 29 10:40:10.720418 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.720359    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scn9v\" (UniqueName: \"kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v\") pod \"6159ef75-c8ac-4cc6-ac35-880ac8279972\" (UID: \"6159ef75-c8ac-4cc6-ac35-880ac8279972\") "
Jul 29 10:40:10.722248 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.722220    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v" (OuterVolumeSpecName: "kube-api-access-scn9v") pod "6159ef75-c8ac-4cc6-ac35-880ac8279972" (UID: "6159ef75-c8ac-4cc6-ac35-880ac8279972"). InnerVolumeSpecName "kube-api-access-scn9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:40:10.743581 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.743554    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache" (OuterVolumeSpecName: "model-cache") pod "6159ef75-c8ac-4cc6-ac35-880ac8279972" (UID: "6159ef75-c8ac-4cc6-ac35-880ac8279972"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:40:10.821523 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.821467    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scn9v\" (UniqueName: \"kubernetes.io/projected/6159ef75-c8ac-4cc6-ac35-880ac8279972-kube-api-access-scn9v\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:40:10.821523 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:10.821487    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6159ef75-c8ac-4cc6-ac35-880ac8279972-model-cache\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:40:11.509782 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:11.509749    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9pt74" event={"ID":"6159ef75-c8ac-4cc6-ac35-880ac8279972","Type":"ContainerDied","Data":"aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85"}
Jul 29 10:40:11.509923 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:11.509786    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec29b34290b21af31bf2eceda1f68c1e35092120908b4f6f87d836fbbac8b85"
Jul 29 10:40:11.509923 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:11.509791    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9pt74"
Jul 29 10:40:23.248952 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.248913    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:40:23.249429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.249245    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerName="download-hf-model"
Jul 29 10:40:23.249429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.249255    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerName="download-hf-model"
Jul 29 10:40:23.249429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.249274    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerName="s3-tls-init-serving"
Jul 29 10:40:23.249429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.249279    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerName="s3-tls-init-serving"
Jul 29 10:40:23.249429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.249341    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6159ef75-c8ac-4cc6-ac35-880ac8279972" containerName="s3-tls-init-serving"
Jul 29 10:40:23.254928 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.254906    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.257321 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257295    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 29 10:40:23.257441 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257354    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:40:23.257507 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257455    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:40:23.257611 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257593    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 29 10:40:23.257664 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257651    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:40:23.257899 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.257880    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:40:23.258052 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.258033    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:40:23.263272 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.263250    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:40:23.426099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.426065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.426099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.426101    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.426335 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.426276    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdnp\" (UniqueName: \"kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.426335 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.426313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.426408 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.426375    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.527607 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.527607 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527583    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.527607 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.527820 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527647    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdnp\" (UniqueName: \"kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.527820 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527671    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.528020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.527998    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.528343 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.528319    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.528439 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.528354    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.530097 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.530077    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.535906 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.535876    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdnp\" (UniqueName: \"kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp\") pod \"isvc-sklearn-batcher-predictor-f9f994b87-dmtd2\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.566244 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.566223    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:23.697642 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:23.697618    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:40:23.699744 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:40:23.699719    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bffa07_f9f7_45b1_9de7_ea44b3e14aa8.slice/crio-707b91c3bede1b189ce92a70f729be7b3d8c701f65059639ddc046cf4825dda4 WatchSource:0}: Error finding container 707b91c3bede1b189ce92a70f729be7b3d8c701f65059639ddc046cf4825dda4: Status 404 returned error can't find the container with id 707b91c3bede1b189ce92a70f729be7b3d8c701f65059639ddc046cf4825dda4
Jul 29 10:40:24.562670 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:24.562636    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerStarted","Data":"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"}
Jul 29 10:40:24.563042 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:24.562703    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerStarted","Data":"707b91c3bede1b189ce92a70f729be7b3d8c701f65059639ddc046cf4825dda4"}
Jul 29 10:40:27.576691 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:27.576657    2551 generic.go:356] "Generic (PLEG): container finished" podID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerID="3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3" exitCode=0
Jul 29 10:40:27.577166 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:27.576712    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerDied","Data":"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"}
Jul 29 10:40:41.650578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:41.650536    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerStarted","Data":"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"}
Jul 29 10:40:43.664102 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:43.664064    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerStarted","Data":"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"}
Jul 29 10:40:45.676488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:45.676453    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerStarted","Data":"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"}
Jul 29 10:40:45.676973 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:45.676719    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:45.676973 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:45.676751    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:45.678207 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:45.678166    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:40:45.698499 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:45.698454    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podStartSLOduration=1.9865603969999999 podStartE2EDuration="22.69844018s" totalImagesPullingTime="17.052958893s" totalInitContainerRuntime="3.65892089s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:40:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:40:45.69586343 +0000 UTC m=+697.598091840" watchObservedRunningTime="2026-07-29 10:40:45.69844018 +0000 UTC m=+697.600668575"
Jul 29 10:40:46.680789 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:46.680753    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:46.681271 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:46.680915    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:40:46.682053 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:46.682022    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:47.685059 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:47.685019    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:40:47.685554 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:47.685504    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:47.689079 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:47.689059    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:40:48.689514 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:48.689475    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:40:48.689925 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:48.689732    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:40:58.690098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:58.690056    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:40:58.690696 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:40:58.690672    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:08.689352 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:08.689309    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:41:08.689741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:08.689718    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:18.689430 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:18.689387    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:41:18.689900 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:18.689876    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:28.689570 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:28.689520    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:41:28.689979 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:28.689915    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:38.689476 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:38.689429    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:41:38.690040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:38.689847    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:48.690118 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:48.690078    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:41:48.690565 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:48.690483    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:41:58.690803 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:58.690766    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:41:58.691318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:41:58.691090    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:08.445364 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.445335    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:42:08.445744 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.445611    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" containerID="cri-o://0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a" gracePeriod=30
Jul 29 10:42:08.445744 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.445641    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" containerID="cri-o://ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6" gracePeriod=30
Jul 29 10:42:08.445839 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.445640    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" containerID="cri-o://a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49" gracePeriod=30
Jul 29 10:42:08.475047 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.475017    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:42:08.478183 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.478169    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.480782 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.480764    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 29 10:42:08.480782 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.480763    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 29 10:42:08.488921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.488900    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:42:08.578821 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.578798    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.578918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.578835    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.578983 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.578928    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tqj\" (UniqueName: \"kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.579055 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.579034    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.579097 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.579064    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680095 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680067    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680174    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680274 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tqj\" (UniqueName: \"kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680535 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.680818 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.680802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.683003 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.682984    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 29 10:42:08.683060 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.683032    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 29 10:42:08.689403 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.689373    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:42:08.689650 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.689623    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tqj\" (UniqueName: \"kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.689771 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.689738    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:08.691176 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.691157    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.692730 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.692709    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.788337 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.788305    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:08.923793 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:08.923765    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:42:08.926550 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:42:08.926520    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ce3639_68b8_4c27_8764_048343454be9.slice/crio-5b2cfb94f019a487e016bfd50f602c29533aad9f5b0ec3adecd76289ff09c13f WatchSource:0}: Error finding container 5b2cfb94f019a487e016bfd50f602c29533aad9f5b0ec3adecd76289ff09c13f: Status 404 returned error can't find the container with id 5b2cfb94f019a487e016bfd50f602c29533aad9f5b0ec3adecd76289ff09c13f
Jul 29 10:42:09.025535 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:09.025451    2551 generic.go:356] "Generic (PLEG): container finished" podID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerID="a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49" exitCode=2
Jul 29 10:42:09.025680 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:09.025524    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerDied","Data":"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"}
Jul 29 10:42:09.027141 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:09.027103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerStarted","Data":"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"}
Jul 29 10:42:09.027243 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:09.027181    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerStarted","Data":"5b2cfb94f019a487e016bfd50f602c29533aad9f5b0ec3adecd76289ff09c13f"}
Jul 29 10:42:12.685770 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:12.685723    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:13.048154 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:13.048102    2551 generic.go:356] "Generic (PLEG): container finished" podID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerID="0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a" exitCode=0
Jul 29 10:42:13.048291 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:13.048172    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerDied","Data":"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"}
Jul 29 10:42:13.049719 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:13.049699    2551 generic.go:356] "Generic (PLEG): container finished" podID="75ce3639-68b8-4c27-8764-048343454be9" containerID="921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa" exitCode=0
Jul 29 10:42:13.049826 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:13.049751    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerDied","Data":"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"}
Jul 29 10:42:14.057314 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057284    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerStarted","Data":"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"}
Jul 29 10:42:14.057746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerStarted","Data":"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"}
Jul 29 10:42:14.057746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057390    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerStarted","Data":"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"}
Jul 29 10:42:14.057746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057696    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:14.057882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057762    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:14.057882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.057802    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:14.059532 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.059503    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:14.060155 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.060110    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:14.079352 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:14.079309    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podStartSLOduration=2.42579354 podStartE2EDuration="6.079296691s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653503151s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:42:14.076763724 +0000 UTC m=+785.978992142" watchObservedRunningTime="2026-07-29 10:42:14.079296691 +0000 UTC m=+785.981525087"
Jul 29 10:42:15.061673 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:15.061628    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:15.062175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:15.062150    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:17.685492 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:17.685451    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:18.689641 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:18.689598    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:42:18.690111 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:18.690088    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:20.066400 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:20.066372    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:42:20.066986 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:20.066954    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:20.067300 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:20.067276    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:22.686029 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:22.685985    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:22.686465 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:22.686152    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:27.685993 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:27.685943    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:28.689756 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:28.689707    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 29 10:42:28.690236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:28.689882    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:28.690236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:28.690021    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:28.690236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:28.690145    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:30.067810 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:30.067725    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:30.068270 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:30.068246    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:32.685661 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:32.685610    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:37.685863 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:37.685825    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 29 10:42:38.583615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.583590    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:38.655902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.655871    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert\") pod \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") "
Jul 29 10:42:38.655902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.655903    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location\") pod \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") "
Jul 29 10:42:38.656154 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.655936    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hdnp\" (UniqueName: \"kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp\") pod \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") "
Jul 29 10:42:38.656154 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.655953    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls\") pod \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") "
Jul 29 10:42:38.656154 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.655983    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config\") pod \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\" (UID: \"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8\") "
Jul 29 10:42:38.656319 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.656248    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" (UID: "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:42:38.656319 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.656290    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" (UID: "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:42:38.656502 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.656478    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" (UID: "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:42:38.658295 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.658264    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" (UID: "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:42:38.658374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.658303    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp" (OuterVolumeSpecName: "kube-api-access-5hdnp") pod "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" (UID: "79bffa07-f9f7-45b1-9de7-ea44b3e14aa8"). InnerVolumeSpecName "kube-api-access-5hdnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:42:38.757088 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.757057    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdnp\" (UniqueName: \"kubernetes.io/projected/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-api-access-5hdnp\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:42:38.757409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.757090    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:42:38.757409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.757105    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:42:38.757409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.757117    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:42:38.757409 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:38.757150    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:42:39.167423 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.167320    2551 generic.go:356] "Generic (PLEG): container finished" podID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerID="ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6" exitCode=0
Jul 29 10:42:39.167423 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.167376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerDied","Data":"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"}
Jul 29 10:42:39.167423 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.167398    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"
Jul 29 10:42:39.167681 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.167443    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2" event={"ID":"79bffa07-f9f7-45b1-9de7-ea44b3e14aa8","Type":"ContainerDied","Data":"707b91c3bede1b189ce92a70f729be7b3d8c701f65059639ddc046cf4825dda4"}
Jul 29 10:42:39.167681 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.167478    2551 scope.go:117] "RemoveContainer" containerID="ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"
Jul 29 10:42:39.175606 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.175590    2551 scope.go:117] "RemoveContainer" containerID="a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"
Jul 29 10:42:39.183884 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.183858    2551 scope.go:117] "RemoveContainer" containerID="0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"
Jul 29 10:42:39.187330 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.187305    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:42:39.190571 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.190552    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-f9f994b87-dmtd2"]
Jul 29 10:42:39.192307 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.192290    2551 scope.go:117] "RemoveContainer" containerID="3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"
Jul 29 10:42:39.200092 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200076    2551 scope.go:117] "RemoveContainer" containerID="ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"
Jul 29 10:42:39.200389 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:42:39.200370    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6\": container with ID starting with ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6 not found: ID does not exist" containerID="ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"
Jul 29 10:42:39.200450 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200396    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6"} err="failed to get container status \"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6\": rpc error: code = NotFound desc = could not find container \"ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6\": container with ID starting with ddadf63798ea97ac33aa0e0fc247579c5db947fc5940047b9fd67ef91f6e28d6 not found: ID does not exist"
Jul 29 10:42:39.200450 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200412    2551 scope.go:117] "RemoveContainer" containerID="a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"
Jul 29 10:42:39.200667 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:42:39.200646    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49\": container with ID starting with a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49 not found: ID does not exist" containerID="a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"
Jul 29 10:42:39.200713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200674    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49"} err="failed to get container status \"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49\": rpc error: code = NotFound desc = could not find container \"a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49\": container with ID starting with a87700dd7b62be710924168e3c786351781d807406c86f742e3dd633d8a5aa49 not found: ID does not exist"
Jul 29 10:42:39.200713 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200692    2551 scope.go:117] "RemoveContainer" containerID="0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"
Jul 29 10:42:39.200958 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:42:39.200939    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a\": container with ID starting with 0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a not found: ID does not exist" containerID="0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"
Jul 29 10:42:39.201020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200969    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a"} err="failed to get container status \"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a\": rpc error: code = NotFound desc = could not find container \"0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a\": container with ID starting with 0ca163d46ef9895c3f757a668019198f108f0087f071117555a7b4d19c50658a not found: ID does not exist"
Jul 29 10:42:39.201020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.200992    2551 scope.go:117] "RemoveContainer" containerID="3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"
Jul 29 10:42:39.201292 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:42:39.201277    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3\": container with ID starting with 3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3 not found: ID does not exist" containerID="3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"
Jul 29 10:42:39.201347 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:39.201296    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3"} err="failed to get container status \"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3\": rpc error: code = NotFound desc = could not find container \"3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3\": container with ID starting with 3010f7cf816394a4e812909987c5187964ccbd02b8322e42b8234e58c039f0a3 not found: ID does not exist"
Jul 29 10:42:40.067420 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:40.067361    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:40.067779 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:40.067724    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:42:40.728901 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:40.728870    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" path="/var/lib/kubelet/pods/79bffa07-f9f7-45b1-9de7-ea44b3e14aa8/volumes"
Jul 29 10:42:50.067024 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:50.066982    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:42:50.067537 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:42:50.067492    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:43:00.067400 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:00.067358    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:43:00.067955 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:00.067732    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:43:10.067162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:10.067111    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:43:10.067587 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:10.067567    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:43:20.067362 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:20.067316    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:43:20.067782 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:20.067757    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:43:29.723994 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:29.723964    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:43:29.724388 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:29.724227    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:43:43.599617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.599586    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.599974    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="storage-initializer"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.599987    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="storage-initializer"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600005    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600011    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600020    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600025    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600036    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent"
Jul 29 10:43:43.600062 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600045    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent"
Jul 29 10:43:43.600368 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600101    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kserve-container"
Jul 29 10:43:43.600368 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600111    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="kube-rbac-proxy"
Jul 29 10:43:43.600368 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.600121    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bffa07-f9f7-45b1-9de7-ea44b3e14aa8" containerName="agent"
Jul 29 10:43:43.603442 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.603422    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.605709 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.605686    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 29 10:43:43.605849 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.605832    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 29 10:43:43.612557 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.612532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:43:43.619530 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.619506    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:43:43.619763 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.619741    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" containerID="cri-o://549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb" gracePeriod=30
Jul 29 10:43:43.619850 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.619829    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" containerID="cri-o://cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5" gracePeriod=30
Jul 29 10:43:43.619927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.619822    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" containerID="cri-o://271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e" gracePeriod=30
Jul 29 10:43:43.690749 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.690724    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.690864 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.690757    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2n5\" (UniqueName: \"kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.690864 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.690787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.791431 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.791404    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.791550 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.791443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2n5\" (UniqueName: \"kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.791550 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.791473    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.792036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.792014    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.793637 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.793616    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.800014 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.799994    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2n5\" (UniqueName: \"kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5\") pod \"message-dumper-predictor-6575bcfdfd-b55v7\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:43.914233 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:43.914183    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:44.044211 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:44.044185    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:43:44.046070 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:43:44.046047    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba0d610_7535_46ae_8dcf_6a49aa679804.slice/crio-b835b2574f6584fd2cd317cfeab3caec3c24fbc95eaef189aef6caadab579d6c WatchSource:0}: Error finding container b835b2574f6584fd2cd317cfeab3caec3c24fbc95eaef189aef6caadab579d6c: Status 404 returned error can't find the container with id b835b2574f6584fd2cd317cfeab3caec3c24fbc95eaef189aef6caadab579d6c
Jul 29 10:43:44.047946 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:44.047932    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:43:44.450240 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:44.450212    2551 generic.go:356] "Generic (PLEG): container finished" podID="75ce3639-68b8-4c27-8764-048343454be9" containerID="cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5" exitCode=2
Jul 29 10:43:44.450399 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:44.450280    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerDied","Data":"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"}
Jul 29 10:43:44.451349 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:44.451328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerStarted","Data":"b835b2574f6584fd2cd317cfeab3caec3c24fbc95eaef189aef6caadab579d6c"}
Jul 29 10:43:45.062336 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:45.062295    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:43:46.461612 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.461578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerStarted","Data":"9cba51c05640baec1e7aa25262553033b970306b48e19d6147ca62bc6f7dcd0d"}
Jul 29 10:43:46.462037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.461640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerStarted","Data":"21ef092eda879d782b2b26612b14a3d7796778877cb334eb0f657938c3b00c2b"}
Jul 29 10:43:46.462037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.461667    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:46.462037 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.461687    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:46.463642 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.463619    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:46.480746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:46.480704    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" podStartSLOduration=2.035133064 podStartE2EDuration="3.480690124s" totalImagesPullingTime="1.44555706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:43:46.479737867 +0000 UTC m=+878.381966279" watchObservedRunningTime="2026-07-29 10:43:46.480690124 +0000 UTC m=+878.382918533"
Jul 29 10:43:48.474491 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:48.474456    2551 generic.go:356] "Generic (PLEG): container finished" podID="75ce3639-68b8-4c27-8764-048343454be9" containerID="549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb" exitCode=0
Jul 29 10:43:48.474999 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:48.474538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerDied","Data":"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"}
Jul 29 10:43:49.723573 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:49.723532    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:43:49.724042 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:49.723898    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:43:50.062298 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:50.062223    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:43:53.480016 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:53.479983    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:43:55.062670 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:55.062628    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:43:55.063060 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:55.062758    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:43:59.723701 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:59.723608    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:43:59.724046 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:43:59.723919    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:00.061952 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:00.061857    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:44:03.644995 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.644961    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:44:03.651459 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.649384    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.652733 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.652630    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 29 10:44:03.652881 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.652709    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 29 10:44:03.659170 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.659147    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:44:03.736738 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.736716    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.736851 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.736746    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.736851 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.736783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.736931 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.736841    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwmg\" (UniqueName: \"kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.736931 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.736884    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.837541 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.837519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.837639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.837548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwmg\" (UniqueName: \"kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.837639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.837569    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.837639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.837607    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.837804 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.837718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.838028 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.838007    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.838246 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.838223    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.838354 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.838285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.840026 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.840000    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.846013 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.845993    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwmg\" (UniqueName: \"kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg\") pod \"isvc-logger-predictor-57c974c7f6-fx9lq\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:03.962334 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:03.962306    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:04.091828 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:04.091805    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:44:04.093594 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:44:04.093570    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bef999_94ce_41a6_9839_b2130e9ea08d.slice/crio-4ca6dfd04fb8429028c3fbe3d3fbf4b4648a78af208defbae159618488103671 WatchSource:0}: Error finding container 4ca6dfd04fb8429028c3fbe3d3fbf4b4648a78af208defbae159618488103671: Status 404 returned error can't find the container with id 4ca6dfd04fb8429028c3fbe3d3fbf4b4648a78af208defbae159618488103671
Jul 29 10:44:04.550090 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:04.550048    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerStarted","Data":"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"}
Jul 29 10:44:04.550283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:04.550120    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerStarted","Data":"4ca6dfd04fb8429028c3fbe3d3fbf4b4648a78af208defbae159618488103671"}
Jul 29 10:44:05.062565 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:05.062526    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:44:08.570741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:08.570708    2551 generic.go:356] "Generic (PLEG): container finished" podID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerID="2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650" exitCode=0
Jul 29 10:44:08.571121 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:08.570776    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerDied","Data":"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"}
Jul 29 10:44:08.601036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:08.601012    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:44:08.602108 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:08.602080    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:44:09.578874 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.578842    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerStarted","Data":"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"}
Jul 29 10:44:09.579242 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.578905    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerStarted","Data":"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"}
Jul 29 10:44:09.579242 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.578932    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerStarted","Data":"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"}
Jul 29 10:44:09.579242 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.579214    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:09.579368 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.579349    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:09.580554 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.580530    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:09.604281 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.604243    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podStartSLOduration=2.872565769 podStartE2EDuration="6.60423087s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.731665101s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:44:09.603030726 +0000 UTC m=+901.505259123" watchObservedRunningTime="2026-07-29 10:44:09.60423087 +0000 UTC m=+901.506459268"
Jul 29 10:44:09.723791 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.723745    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Jul 29 10:44:09.723953 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.723884    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:44:09.724072 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.724048    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:09.724199 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:09.724186    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:44:10.062243 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:10.062208    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 29 10:44:10.583187 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:10.583156    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:10.583619 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:10.583317    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:10.584322 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:10.584293    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:11.587469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:11.587425    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:11.587985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:11.587952    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:13.772311 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.772286    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:44:13.811784 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.811750    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert\") pod \"75ce3639-68b8-4c27-8764-048343454be9\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") "
Jul 29 10:44:13.811908 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.811802    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55tqj\" (UniqueName: \"kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj\") pod \"75ce3639-68b8-4c27-8764-048343454be9\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") "
Jul 29 10:44:13.811908 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.811863    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location\") pod \"75ce3639-68b8-4c27-8764-048343454be9\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") "
Jul 29 10:44:13.811999 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.811936    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls\") pod \"75ce3639-68b8-4c27-8764-048343454be9\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") "
Jul 29 10:44:13.811999 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.811977    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config\") pod \"75ce3639-68b8-4c27-8764-048343454be9\" (UID: \"75ce3639-68b8-4c27-8764-048343454be9\") "
Jul 29 10:44:13.812164 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.812118    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75ce3639-68b8-4c27-8764-048343454be9" (UID: "75ce3639-68b8-4c27-8764-048343454be9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:44:13.812255 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.812163    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75ce3639-68b8-4c27-8764-048343454be9" (UID: "75ce3639-68b8-4c27-8764-048343454be9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:13.812343 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.812324    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:44:13.812402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.812347    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75ce3639-68b8-4c27-8764-048343454be9-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:44:13.812455 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.812428    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75ce3639-68b8-4c27-8764-048343454be9" (UID: "75ce3639-68b8-4c27-8764-048343454be9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:44:13.813856 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.813829    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj" (OuterVolumeSpecName: "kube-api-access-55tqj") pod "75ce3639-68b8-4c27-8764-048343454be9" (UID: "75ce3639-68b8-4c27-8764-048343454be9"). InnerVolumeSpecName "kube-api-access-55tqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:13.813929 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.813886    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75ce3639-68b8-4c27-8764-048343454be9" (UID: "75ce3639-68b8-4c27-8764-048343454be9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:44:13.913617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.913556    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75ce3639-68b8-4c27-8764-048343454be9-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:44:13.913617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.913580    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75ce3639-68b8-4c27-8764-048343454be9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:44:13.913617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:13.913590    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55tqj\" (UniqueName: \"kubernetes.io/projected/75ce3639-68b8-4c27-8764-048343454be9-kube-api-access-55tqj\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:44:14.602379 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.602343    2551 generic.go:356] "Generic (PLEG): container finished" podID="75ce3639-68b8-4c27-8764-048343454be9" containerID="271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e" exitCode=0
Jul 29 10:44:14.602547 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.602413    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"
Jul 29 10:44:14.602547 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.602424    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerDied","Data":"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"}
Jul 29 10:44:14.602547 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.602492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5" event={"ID":"75ce3639-68b8-4c27-8764-048343454be9","Type":"ContainerDied","Data":"5b2cfb94f019a487e016bfd50f602c29533aad9f5b0ec3adecd76289ff09c13f"}
Jul 29 10:44:14.602547 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.602523    2551 scope.go:117] "RemoveContainer" containerID="271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"
Jul 29 10:44:14.611747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.611727    2551 scope.go:117] "RemoveContainer" containerID="cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"
Jul 29 10:44:14.618875 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.618858    2551 scope.go:117] "RemoveContainer" containerID="549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"
Jul 29 10:44:14.625478 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.625457    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:44:14.626667 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.626511    2551 scope.go:117] "RemoveContainer" containerID="921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"
Jul 29 10:44:14.629342 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.629319    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7fb595d89d-r7vn5"]
Jul 29 10:44:14.634079 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634062    2551 scope.go:117] "RemoveContainer" containerID="271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"
Jul 29 10:44:14.634365 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:44:14.634346    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e\": container with ID starting with 271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e not found: ID does not exist" containerID="271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"
Jul 29 10:44:14.634429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634371    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e"} err="failed to get container status \"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e\": rpc error: code = NotFound desc = could not find container \"271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e\": container with ID starting with 271d8433f2efd545f4217117a1be097111a37561dd3429f09168a3cc8bd1d56e not found: ID does not exist"
Jul 29 10:44:14.634429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634389    2551 scope.go:117] "RemoveContainer" containerID="cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"
Jul 29 10:44:14.634639 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:44:14.634623    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5\": container with ID starting with cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5 not found: ID does not exist" containerID="cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"
Jul 29 10:44:14.634680 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634648    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5"} err="failed to get container status \"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5\": rpc error: code = NotFound desc = could not find container \"cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5\": container with ID starting with cfd11e71366f55bf97f152cd1513674e28769270c25a85d3c8039d34855a34d5 not found: ID does not exist"
Jul 29 10:44:14.634680 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634666    2551 scope.go:117] "RemoveContainer" containerID="549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"
Jul 29 10:44:14.634897 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:44:14.634880    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb\": container with ID starting with 549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb not found: ID does not exist" containerID="549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"
Jul 29 10:44:14.634933 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634902    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb"} err="failed to get container status \"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb\": rpc error: code = NotFound desc = could not find container \"549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb\": container with ID starting with 549612e682d28c0d1ed52caa032fa6f05c635088793cdd81a111374f233a02eb not found: ID does not exist"
Jul 29 10:44:14.634933 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.634924    2551 scope.go:117] "RemoveContainer" containerID="921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"
Jul 29 10:44:14.635142 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:44:14.635113    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa\": container with ID starting with 921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa not found: ID does not exist" containerID="921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"
Jul 29 10:44:14.635190 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.635148    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa"} err="failed to get container status \"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa\": rpc error: code = NotFound desc = could not find container \"921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa\": container with ID starting with 921b9350d478408bc029c3e4f40f92d6a8261eb8b92e49ed50401da16bed27fa not found: ID does not exist"
Jul 29 10:44:14.727701 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:14.727671    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ce3639-68b8-4c27-8764-048343454be9" path="/var/lib/kubelet/pods/75ce3639-68b8-4c27-8764-048343454be9/volumes"
Jul 29 10:44:16.591338 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:16.591307    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:44:16.591989 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:16.591947    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:16.592429 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:16.592406    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:26.592172 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:26.592118    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:26.592604 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:26.592570    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:36.591896 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:36.591848    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:36.592442 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:36.592416    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:46.592636 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:46.592592    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:46.593175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:46.593069    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:44:56.591829 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:56.591789    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:44:56.592249 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:44:56.592188    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:06.592071 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:06.592025    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:45:06.594301 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:06.592709    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:16.592502 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:16.592458    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:45:16.592924 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:16.592890    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:22.728615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:22.728583    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:22.729226 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:22.728697    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:28.637806 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.637778    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-b55v7_dba0d610-7535-46ae-8dcf-6a49aa679804/kserve-container/0.log"
Jul 29 10:45:28.815189 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.815163    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:45:28.815431 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.815388    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kserve-container" containerID="cri-o://21ef092eda879d782b2b26612b14a3d7796778877cb334eb0f657938c3b00c2b" gracePeriod=30
Jul 29 10:45:28.815527 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.815433    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kube-rbac-proxy" containerID="cri-o://9cba51c05640baec1e7aa25262553033b970306b48e19d6147ca62bc6f7dcd0d" gracePeriod=30
Jul 29 10:45:28.877945 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.877916    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:45:28.878284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.878256    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" containerID="cri-o://bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0" gracePeriod=30
Jul 29 10:45:28.878414 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.878393    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" containerID="cri-o://3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa" gracePeriod=30
Jul 29 10:45:28.878497 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.878480    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" containerID="cri-o://9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3" gracePeriod=30
Jul 29 10:45:28.913422 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913397    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:45:28.913761 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913748    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container"
Jul 29 10:45:28.913807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913763    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container"
Jul 29 10:45:28.913807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913772    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent"
Jul 29 10:45:28.913807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913777    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent"
Jul 29 10:45:28.913807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913792    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="storage-initializer"
Jul 29 10:45:28.913807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913800    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="storage-initializer"
Jul 29 10:45:28.913958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913811    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy"
Jul 29 10:45:28.913958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913817    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy"
Jul 29 10:45:28.913958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913890    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="agent"
Jul 29 10:45:28.913958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913924    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kserve-container"
Jul 29 10:45:28.913958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.913932    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ce3639-68b8-4c27-8764-048343454be9" containerName="kube-rbac-proxy"
Jul 29 10:45:28.916929 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.916910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:28.919460 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.919436    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 29 10:45:28.919674 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.919658    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 29 10:45:28.927754 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.927735    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:45:28.949493 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.949470    2551 generic.go:356] "Generic (PLEG): container finished" podID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerID="9cba51c05640baec1e7aa25262553033b970306b48e19d6147ca62bc6f7dcd0d" exitCode=2
Jul 29 10:45:28.949493 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.949493    2551 generic.go:356] "Generic (PLEG): container finished" podID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerID="21ef092eda879d782b2b26612b14a3d7796778877cb334eb0f657938c3b00c2b" exitCode=2
Jul 29 10:45:28.949624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.949517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerDied","Data":"9cba51c05640baec1e7aa25262553033b970306b48e19d6147ca62bc6f7dcd0d"}
Jul 29 10:45:28.949624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:28.949555    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerDied","Data":"21ef092eda879d782b2b26612b14a3d7796778877cb334eb0f657938c3b00c2b"}
Jul 29 10:45:29.000958 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.000932    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.001066 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.000992    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.001066 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.001037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.001231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.001122    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.001231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.001161    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjgs\" (UniqueName: \"kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.101795 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.101770    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.101898 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.101826    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.101898 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.101881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.102100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.101909    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjgs\" (UniqueName: \"kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.102100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.101946    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.102100 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:29.102029    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 29 10:45:29.102380 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:29.102106    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls podName:0c921ba4-9e44-4e84-9446-7b208103a67c nodeName:}" failed. No retries permitted until 2026-07-29 10:45:29.602082254 +0000 UTC m=+981.504310648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-fvxm7" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 29 10:45:29.102380 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.102363    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.102639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.102584    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.102639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.102584    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.110145 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.110107    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjgs\" (UniqueName: \"kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.117180 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.117163    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:45:29.202803 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.202768    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p2n5\" (UniqueName: \"kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5\") pod \"dba0d610-7535-46ae-8dcf-6a49aa679804\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") "
Jul 29 10:45:29.202963 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.202822    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls\") pod \"dba0d610-7535-46ae-8dcf-6a49aa679804\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") "
Jul 29 10:45:29.202963 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.202897    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config\") pod \"dba0d610-7535-46ae-8dcf-6a49aa679804\" (UID: \"dba0d610-7535-46ae-8dcf-6a49aa679804\") "
Jul 29 10:45:29.203306 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.203283    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dba0d610-7535-46ae-8dcf-6a49aa679804" (UID: "dba0d610-7535-46ae-8dcf-6a49aa679804"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:29.204941 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.204916    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dba0d610-7535-46ae-8dcf-6a49aa679804" (UID: "dba0d610-7535-46ae-8dcf-6a49aa679804"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:45:29.205040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.204958    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5" (OuterVolumeSpecName: "kube-api-access-8p2n5") pod "dba0d610-7535-46ae-8dcf-6a49aa679804" (UID: "dba0d610-7535-46ae-8dcf-6a49aa679804"). InnerVolumeSpecName "kube-api-access-8p2n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:45:29.303933 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.303908    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:29.303933 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.303930    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2n5\" (UniqueName: \"kubernetes.io/projected/dba0d610-7535-46ae-8dcf-6a49aa679804-kube-api-access-8p2n5\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:29.303933 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.303940    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dba0d610-7535-46ae-8dcf-6a49aa679804-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:29.605808 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.605724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.608100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.608079    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxm7\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.827985 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.827959    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:45:29.955776 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.955748    2551 generic.go:356] "Generic (PLEG): container finished" podID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerID="9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3" exitCode=2
Jul 29 10:45:29.955932 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.955820    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerDied","Data":"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"}
Jul 29 10:45:29.957193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.957171    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7" event={"ID":"dba0d610-7535-46ae-8dcf-6a49aa679804","Type":"ContainerDied","Data":"b835b2574f6584fd2cd317cfeab3caec3c24fbc95eaef189aef6caadab579d6c"}
Jul 29 10:45:29.957307 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.957204    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"
Jul 29 10:45:29.957307 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.957235    2551 scope.go:117] "RemoveContainer" containerID="9cba51c05640baec1e7aa25262553033b970306b48e19d6147ca62bc6f7dcd0d"
Jul 29 10:45:29.961283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.961247    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:45:29.974120 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.974101    2551 scope.go:117] "RemoveContainer" containerID="21ef092eda879d782b2b26612b14a3d7796778877cb334eb0f657938c3b00c2b"
Jul 29 10:45:29.987839 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.987814    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:45:29.991580 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:29.991560    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b55v7"]
Jul 29 10:45:30.728579 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:30.728542    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" path="/var/lib/kubelet/pods/dba0d610-7535-46ae-8dcf-6a49aa679804/volumes"
Jul 29 10:45:30.963084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:30.963045    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerStarted","Data":"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"}
Jul 29 10:45:30.963496 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:30.963118    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerStarted","Data":"18d2ed4047c7b92a817460a5917bb0a5430ddfd526f836c70c97be62063ad820"}
Jul 29 10:45:31.588580 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:31.588541    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:32.723744 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:32.723712    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:45:32.724239 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:32.724210    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:32.976038 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:32.976010    2551 generic.go:356] "Generic (PLEG): container finished" podID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerID="bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0" exitCode=0
Jul 29 10:45:32.976164 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:32.976077    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerDied","Data":"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"}
Jul 29 10:45:36.588538 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:36.588499    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:41.588022 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:41.587977    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:41.588472 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:41.588115    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:42.723369 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:42.723319    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:45:42.723771 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:42.723589    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:46.587822 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:46.587777    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:51.588326 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:51.588281    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:52.724028 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:52.723973    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 29 10:45:52.724468 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:52.724405    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:45:52.728259 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:52.728237    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:52.728389 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:52.728334    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:56.588442 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:56.588394    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 29 10:45:59.061156 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.061117    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:59.104262 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.104239    2551 generic.go:356] "Generic (PLEG): container finished" podID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerID="3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa" exitCode=137
Jul 29 10:45:59.104374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.104295    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"
Jul 29 10:45:59.104374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.104319    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerDied","Data":"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"}
Jul 29 10:45:59.104491 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.104395    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq" event={"ID":"54bef999-94ce-41a6-9839-b2130e9ea08d","Type":"ContainerDied","Data":"4ca6dfd04fb8429028c3fbe3d3fbf4b4648a78af208defbae159618488103671"}
Jul 29 10:45:59.104491 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.104443    2551 scope.go:117] "RemoveContainer" containerID="3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"
Jul 29 10:45:59.112418 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.112403    2551 scope.go:117] "RemoveContainer" containerID="9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"
Jul 29 10:45:59.119098 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.119081    2551 scope.go:117] "RemoveContainer" containerID="bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"
Jul 29 10:45:59.119979 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.119965    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert\") pod \"54bef999-94ce-41a6-9839-b2130e9ea08d\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") "
Jul 29 10:45:59.120040 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120023    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhwmg\" (UniqueName: \"kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg\") pod \"54bef999-94ce-41a6-9839-b2130e9ea08d\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") "
Jul 29 10:45:59.120099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120054    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location\") pod \"54bef999-94ce-41a6-9839-b2130e9ea08d\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") "
Jul 29 10:45:59.120099 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120074    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls\") pod \"54bef999-94ce-41a6-9839-b2130e9ea08d\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") "
Jul 29 10:45:59.120220 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120113    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config\") pod \"54bef999-94ce-41a6-9839-b2130e9ea08d\" (UID: \"54bef999-94ce-41a6-9839-b2130e9ea08d\") "
Jul 29 10:45:59.120446 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120420    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54bef999-94ce-41a6-9839-b2130e9ea08d" (UID: "54bef999-94ce-41a6-9839-b2130e9ea08d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:59.120648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120448    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54bef999-94ce-41a6-9839-b2130e9ea08d" (UID: "54bef999-94ce-41a6-9839-b2130e9ea08d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:45:59.120760 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.120709    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54bef999-94ce-41a6-9839-b2130e9ea08d" (UID: "54bef999-94ce-41a6-9839-b2130e9ea08d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:45:59.121955 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.121932    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54bef999-94ce-41a6-9839-b2130e9ea08d" (UID: "54bef999-94ce-41a6-9839-b2130e9ea08d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:45:59.122067 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.122050    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg" (OuterVolumeSpecName: "kube-api-access-nhwmg") pod "54bef999-94ce-41a6-9839-b2130e9ea08d" (UID: "54bef999-94ce-41a6-9839-b2130e9ea08d"). InnerVolumeSpecName "kube-api-access-nhwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:45:59.135354 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.135335    2551 scope.go:117] "RemoveContainer" containerID="2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"
Jul 29 10:45:59.142167 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142147    2551 scope.go:117] "RemoveContainer" containerID="3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"
Jul 29 10:45:59.142415 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:59.142396    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa\": container with ID starting with 3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa not found: ID does not exist" containerID="3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"
Jul 29 10:45:59.142474 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142421    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa"} err="failed to get container status \"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa\": rpc error: code = NotFound desc = could not find container \"3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa\": container with ID starting with 3c62fc4775a23070402e461e135882c472efef1dd980ae96e695427a84496eaa not found: ID does not exist"
Jul 29 10:45:59.142474 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142437    2551 scope.go:117] "RemoveContainer" containerID="9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"
Jul 29 10:45:59.142662 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:59.142645    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3\": container with ID starting with 9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3 not found: ID does not exist" containerID="9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"
Jul 29 10:45:59.142700 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142667    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3"} err="failed to get container status \"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3\": rpc error: code = NotFound desc = could not find container \"9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3\": container with ID starting with 9b87a2ebf328f8549140bb3a10b46971e397e3cb8cec1724ed23b7843ba561f3 not found: ID does not exist"
Jul 29 10:45:59.142700 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142681    2551 scope.go:117] "RemoveContainer" containerID="bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"
Jul 29 10:45:59.142889 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:59.142870    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0\": container with ID starting with bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0 not found: ID does not exist" containerID="bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"
Jul 29 10:45:59.142936 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142891    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0"} err="failed to get container status \"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0\": rpc error: code = NotFound desc = could not find container \"bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0\": container with ID starting with bdf0d75d426ac6a3cdd473d529996e319dd61886febc74db1353ffca0e0f72c0 not found: ID does not exist"
Jul 29 10:45:59.142936 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.142903    2551 scope.go:117] "RemoveContainer" containerID="2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"
Jul 29 10:45:59.143147 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:45:59.143110    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650\": container with ID starting with 2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650 not found: ID does not exist" containerID="2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"
Jul 29 10:45:59.143193 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.143141    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650"} err="failed to get container status \"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650\": rpc error: code = NotFound desc = could not find container \"2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650\": container with ID starting with 2e0a790d22fd0f205aff4b80d455f77680416dc320896a8838b3d17f4dcc9650 not found: ID does not exist"
Jul 29 10:45:59.220894 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.220873    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhwmg\" (UniqueName: \"kubernetes.io/projected/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-api-access-nhwmg\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:59.220894 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.220892    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54bef999-94ce-41a6-9839-b2130e9ea08d-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:59.220998 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.220902    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54bef999-94ce-41a6-9839-b2130e9ea08d-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:59.220998 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.220912    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:59.220998 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.220920    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54bef999-94ce-41a6-9839-b2130e9ea08d-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:45:59.427513 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.427489    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:45:59.431089 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:45:59.431067    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-57c974c7f6-fx9lq"]
Jul 29 10:46:00.111612 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:46:00.111554    2551 generic.go:356] "Generic (PLEG): container finished" podID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerID="c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012" exitCode=0
Jul 29 10:46:00.111612 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:46:00.111595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerDied","Data":"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"}
Jul 29 10:46:00.732303 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:46:00.732273    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" path="/var/lib/kubelet/pods/54bef999-94ce-41a6-9839-b2130e9ea08d/volumes"
Jul 29 10:47:51.639104 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:51.639062    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerStarted","Data":"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"}
Jul 29 10:47:51.639556 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:51.639208    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:47:51.639556 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:51.639249    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerStarted","Data":"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"}
Jul 29 10:47:51.669048 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:51.669000    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podStartSLOduration=3.435097023 podStartE2EDuration="2m23.668986188s" totalImagesPullingTime="1m50.522534244s" totalInitContainerRuntime="29.711354921s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:47:51.667409448 +0000 UTC m=+1123.569637864" watchObservedRunningTime="2026-07-29 10:47:51.668986188 +0000 UTC m=+1123.571214585"
Jul 29 10:47:52.644146 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:52.644096    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:47:52.645364 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:52.645335    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 29 10:47:53.648807 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:53.648761    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 29 10:47:58.653171 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:58.653117    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:47:58.653782 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:47:58.653751    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 29 10:48:08.654283 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:08.654254    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:48:15.052909 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:15.052876    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:48:15.053436 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:15.053111    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container" containerID="cri-o://8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f" gracePeriod=30
Jul 29 10:48:15.053436 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:15.053178    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kube-rbac-proxy" containerID="cri-o://cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec" gracePeriod=30
Jul 29 10:48:15.749715 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:15.749682    2551 generic.go:356] "Generic (PLEG): container finished" podID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerID="cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec" exitCode=2
Jul 29 10:48:15.749884 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:15.749757    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerDied","Data":"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"}
Jul 29 10:48:16.332704 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.332645    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:48:16.454688 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.454660    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location\") pod \"0c921ba4-9e44-4e84-9446-7b208103a67c\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") "
Jul 29 10:48:16.454815 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.454703    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config\") pod \"0c921ba4-9e44-4e84-9446-7b208103a67c\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") "
Jul 29 10:48:16.454815 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.454721    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert\") pod \"0c921ba4-9e44-4e84-9446-7b208103a67c\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") "
Jul 29 10:48:16.454815 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.454748    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") pod \"0c921ba4-9e44-4e84-9446-7b208103a67c\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") "
Jul 29 10:48:16.454815 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.454778    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmjgs\" (UniqueName: \"kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs\") pod \"0c921ba4-9e44-4e84-9446-7b208103a67c\" (UID: \"0c921ba4-9e44-4e84-9446-7b208103a67c\") "
Jul 29 10:48:16.455113 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.455092    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c921ba4-9e44-4e84-9446-7b208103a67c" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:48:16.455219 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.455193    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c921ba4-9e44-4e84-9446-7b208103a67c" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:48:16.456822 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.456794    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c921ba4-9e44-4e84-9446-7b208103a67c" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:48:16.456923 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.456838    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs" (OuterVolumeSpecName: "kube-api-access-jmjgs") pod "0c921ba4-9e44-4e84-9446-7b208103a67c" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c"). InnerVolumeSpecName "kube-api-access-jmjgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:48:16.459608 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.459588    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c921ba4-9e44-4e84-9446-7b208103a67c" (UID: "0c921ba4-9e44-4e84-9446-7b208103a67c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:48:16.556117 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.556094    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:48:16.556117 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.556114    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c921ba4-9e44-4e84-9446-7b208103a67c-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:48:16.556252 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.556144    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c921ba4-9e44-4e84-9446-7b208103a67c-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:48:16.556252 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.556153    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjgs\" (UniqueName: \"kubernetes.io/projected/0c921ba4-9e44-4e84-9446-7b208103a67c-kube-api-access-jmjgs\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:48:16.556252 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.556161    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c921ba4-9e44-4e84-9446-7b208103a67c-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:48:16.755624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.755600    2551 generic.go:356] "Generic (PLEG): container finished" podID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerID="8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f" exitCode=0
Jul 29 10:48:16.755742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.755657    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerDied","Data":"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"}
Jul 29 10:48:16.755742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.755668    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"
Jul 29 10:48:16.755742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.755698    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7" event={"ID":"0c921ba4-9e44-4e84-9446-7b208103a67c","Type":"ContainerDied","Data":"18d2ed4047c7b92a817460a5917bb0a5430ddfd526f836c70c97be62063ad820"}
Jul 29 10:48:16.755742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.755725    2551 scope.go:117] "RemoveContainer" containerID="cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"
Jul 29 10:48:16.764334 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.764308    2551 scope.go:117] "RemoveContainer" containerID="8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"
Jul 29 10:48:16.771747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.771728    2551 scope.go:117] "RemoveContainer" containerID="c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"
Jul 29 10:48:16.774537 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.774515    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:48:16.780101 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780083    2551 scope.go:117] "RemoveContainer" containerID="cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"
Jul 29 10:48:16.780208 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780188    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxm7"]
Jul 29 10:48:16.780416 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:48:16.780398    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec\": container with ID starting with cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec not found: ID does not exist" containerID="cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"
Jul 29 10:48:16.780470 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780422    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec"} err="failed to get container status \"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec\": rpc error: code = NotFound desc = could not find container \"cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec\": container with ID starting with cd91aebec7c0cf03ae5c735e6e59f513e9d74b38ec69db73bbba9fd8277c6aec not found: ID does not exist"
Jul 29 10:48:16.780470 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780441    2551 scope.go:117] "RemoveContainer" containerID="8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"
Jul 29 10:48:16.780700 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:48:16.780685    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f\": container with ID starting with 8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f not found: ID does not exist" containerID="8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"
Jul 29 10:48:16.780738 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780703    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f"} err="failed to get container status \"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f\": rpc error: code = NotFound desc = could not find container \"8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f\": container with ID starting with 8b5f8c0266add8b81968999a99b58d2b47131a7e740154095c2413585cd6102f not found: ID does not exist"
Jul 29 10:48:16.780738 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780715    2551 scope.go:117] "RemoveContainer" containerID="c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"
Jul 29 10:48:16.780957 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:48:16.780939    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012\": container with ID starting with c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012 not found: ID does not exist" containerID="c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"
Jul 29 10:48:16.780996 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:16.780963    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012"} err="failed to get container status \"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012\": rpc error: code = NotFound desc = could not find container \"c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012\": container with ID starting with c87eb8b7c64c56aa1aa8728311324db56c60399df98ac6e9ccf6c28b49abf012 not found: ID does not exist"
Jul 29 10:48:18.728095 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:18.728064    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" path="/var/lib/kubelet/pods/0c921ba4-9e44-4e84-9446-7b208103a67c/volumes"
Jul 29 10:48:20.207488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207454    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:48:20.207945 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207933    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent"
Jul 29 10:48:20.208017 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207952    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent"
Jul 29 10:48:20.208017 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207976    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="storage-initializer"
Jul 29 10:48:20.208017 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207986    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="storage-initializer"
Jul 29 10:48:20.208017 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.207996    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208017 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208005    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208020    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208029    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208038    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208048    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208069    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208077    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208086    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208094    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208109    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="storage-initializer"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208118    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="storage-initializer"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208144    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208152    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208227    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208240    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208250    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208263    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="kube-rbac-proxy"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208274    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c921ba4-9e44-4e84-9446-7b208103a67c" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208283    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba0d610-7535-46ae-8dcf-6a49aa679804" containerName="kserve-container"
Jul 29 10:48:20.208288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.208291    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bef999-94ce-41a6-9839-b2130e9ea08d" containerName="agent"
Jul 29 10:48:20.241918 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.241893    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:48:20.242031 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.241982    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.244484 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.244437    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:48:20.244879 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.244865    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:48:20.245287 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.245273    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:48:20.245534 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.245517    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 29 10:48:20.245534 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.245523    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 29 10:48:20.245673 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.245542    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:48:20.245673 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.245568    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:48:20.383930 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.383906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.384020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.383936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27rg\" (UniqueName: \"kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.384020 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.383968    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.384094 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.384018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.384094 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.384058    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485339 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485292    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485339 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27rg\" (UniqueName: \"kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485455 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485345    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485455 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485363    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485455 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485803 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485786    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.485924 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.485907    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.486041 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.486024    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.487895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.487877    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.493624 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.493602    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27rg\" (UniqueName: \"kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8js6t\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.551913 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.551893    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:20.690335 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.690310    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:48:20.692497 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:48:20.692471    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbd9a61_f6cd_43f3_9a2f_646e9eadf504.slice/crio-f197949bd17b3c08c4c0bfbcd6fee73099310daa9bd956fa4e4afe4789527aad WatchSource:0}: Error finding container f197949bd17b3c08c4c0bfbcd6fee73099310daa9bd956fa4e4afe4789527aad: Status 404 returned error can't find the container with id f197949bd17b3c08c4c0bfbcd6fee73099310daa9bd956fa4e4afe4789527aad
Jul 29 10:48:20.772377 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:20.772347    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerStarted","Data":"f197949bd17b3c08c4c0bfbcd6fee73099310daa9bd956fa4e4afe4789527aad"}
Jul 29 10:48:21.778291 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:21.778256    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerStarted","Data":"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"}
Jul 29 10:48:49.893980 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:49.893948    2551 generic.go:356] "Generic (PLEG): container finished" podID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerID="e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96" exitCode=0
Jul 29 10:48:49.894351 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:49.893993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerDied","Data":"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"}
Jul 29 10:48:50.899999 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:50.899965    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerStarted","Data":"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"}
Jul 29 10:48:50.900524 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:50.900024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerStarted","Data":"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"}
Jul 29 10:48:50.900524 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:50.900204    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:50.922011 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:50.921959    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" podStartSLOduration=1.9960049949999998 podStartE2EDuration="30.921940581s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.925935586s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:50.919381281 +0000 UTC m=+1182.821609678" watchObservedRunningTime="2026-07-29 10:48:50.921940581 +0000 UTC m=+1182.824168981"
Jul 29 10:48:51.904812 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:51.904772    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:48:57.914796 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:48:57.914769    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:49:08.629557 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:49:08.629528    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:49:08.631056 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:49:08.631035    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:50:24.729205 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:24.729168    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:50:35.766292 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:35.766245    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 10:50:36.301861 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:36.301829    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:50:36.302093 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:36.302047    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kserve-container" containerID="cri-o://d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae" gracePeriod=30
Jul 29 10:50:36.302162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:36.302083    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kube-rbac-proxy" containerID="cri-o://af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a" gracePeriod=30
Jul 29 10:50:37.356115 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.356079    2551 generic.go:356] "Generic (PLEG): container finished" podID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerID="af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a" exitCode=2
Jul 29 10:50:37.356494 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.356157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerDied","Data":"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"}
Jul 29 10:50:37.543373 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.543350    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:50:37.634031 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.633946    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls\") pod \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") "
Jul 29 10:50:37.634031 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634008    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config\") pod \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") "
Jul 29 10:50:37.634273 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634051    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location\") pod \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") "
Jul 29 10:50:37.634273 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634079    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert\") pod \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") "
Jul 29 10:50:37.634273 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634101    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c27rg\" (UniqueName: \"kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg\") pod \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\" (UID: \"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504\") "
Jul 29 10:50:37.634438 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634414    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" (UID: "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:37.634479 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.634456    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" (UID: "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:50:37.636165 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.636121    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" (UID: "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:50:37.636298 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.636276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg" (OuterVolumeSpecName: "kube-api-access-c27rg") pod "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" (UID: "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504"). InnerVolumeSpecName "kube-api-access-c27rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:50:37.638676 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.638652    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" (UID: "5fbd9a61-f6cd-43f3-9a2f-646e9eadf504"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:50:37.735729 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.735702    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:50:37.735729 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.735725    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:50:37.735729 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.735734    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:50:37.735926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.735743    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:50:37.735926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:37.735752    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27rg\" (UniqueName: \"kubernetes.io/projected/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504-kube-api-access-c27rg\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:50:38.362476 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.362439    2551 generic.go:356] "Generic (PLEG): container finished" podID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerID="d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae" exitCode=0
Jul 29 10:50:38.362927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.362501    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"
Jul 29 10:50:38.362927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.362528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerDied","Data":"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"}
Jul 29 10:50:38.362927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.362604    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t" event={"ID":"5fbd9a61-f6cd-43f3-9a2f-646e9eadf504","Type":"ContainerDied","Data":"f197949bd17b3c08c4c0bfbcd6fee73099310daa9bd956fa4e4afe4789527aad"}
Jul 29 10:50:38.362927 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.362634    2551 scope.go:117] "RemoveContainer" containerID="af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"
Jul 29 10:50:38.372615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.372594    2551 scope.go:117] "RemoveContainer" containerID="d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"
Jul 29 10:50:38.380509 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.380492    2551 scope.go:117] "RemoveContainer" containerID="e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"
Jul 29 10:50:38.385383 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.385360    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:50:38.389425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.389405    2551 scope.go:117] "RemoveContainer" containerID="af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"
Jul 29 10:50:38.389681 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.389663    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8js6t"]
Jul 29 10:50:38.389728 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:50:38.389699    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a\": container with ID starting with af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a not found: ID does not exist" containerID="af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"
Jul 29 10:50:38.389765 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.389722    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a"} err="failed to get container status \"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a\": rpc error: code = NotFound desc = could not find container \"af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a\": container with ID starting with af0b031f66c2771616bd3c79ed32cedc9528fef05f0b1abf3cdedbd9b032729a not found: ID does not exist"
Jul 29 10:50:38.389765 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.389740    2551 scope.go:117] "RemoveContainer" containerID="d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"
Jul 29 10:50:38.389985 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:50:38.389964    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae\": container with ID starting with d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae not found: ID does not exist" containerID="d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"
Jul 29 10:50:38.390053 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.389994    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae"} err="failed to get container status \"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae\": rpc error: code = NotFound desc = could not find container \"d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae\": container with ID starting with d9e87e5d499abec99140436347cfecca2b6152ecc9dead139bacfea3dd914fae not found: ID does not exist"
Jul 29 10:50:38.390053 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.390018    2551 scope.go:117] "RemoveContainer" containerID="e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"
Jul 29 10:50:38.390256 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:50:38.390239    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96\": container with ID starting with e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96 not found: ID does not exist" containerID="e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"
Jul 29 10:50:38.390301 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.390262    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96"} err="failed to get container status \"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96\": rpc error: code = NotFound desc = could not find container \"e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96\": container with ID starting with e48069e9dbeb69f76782fdc24781ab01cfbb930d38f018e2d6a2fd26b4900e96 not found: ID does not exist"
Jul 29 10:50:38.727572 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:38.727541    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" path="/var/lib/kubelet/pods/5fbd9a61-f6cd-43f3-9a2f-646e9eadf504/volumes"
Jul 29 10:50:41.473035 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473003    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473343    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kube-rbac-proxy"
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473355    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kube-rbac-proxy"
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473367    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="storage-initializer"
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473372    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="storage-initializer"
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473380    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kserve-container"
Jul 29 10:50:41.473396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473385    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kserve-container"
Jul 29 10:50:41.473578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473430    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kube-rbac-proxy"
Jul 29 10:50:41.473578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.473437    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fbd9a61-f6cd-43f3-9a2f-646e9eadf504" containerName="kserve-container"
Jul 29 10:50:41.476639 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.476621    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.479213 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.479189    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:50:41.479326 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.479188    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:50:41.479376 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.479333    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 29 10:50:41.479376 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.479341    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 29 10:50:41.480310 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.480292    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:50:41.480369 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.480339    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:50:41.480425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.480292    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:50:41.486276 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.486255    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:50:41.664144 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.664100    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghcvf\" (UniqueName: \"kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.664282 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.664147    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.664282 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.664176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.664282 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.664261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.664433 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.664347    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765156 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765067    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765156 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765156    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765194    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghcvf\" (UniqueName: \"kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765214    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765304 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765239    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.765717 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.765689    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.766034 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.766010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.766118 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.766016    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.767756 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.767731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.773532 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.773503    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghcvf\" (UniqueName: \"kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.787227 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.787201    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:50:41.916706 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:41.916684    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:50:41.918697 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:50:41.918667    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34760b6f_bbfa_4dfc_a2eb_af6ce3fdc975.slice/crio-c19ac35cd402974dae9fb0b6b6a4f71ecc0bf10a1483479852956ef87aa69686 WatchSource:0}: Error finding container c19ac35cd402974dae9fb0b6b6a4f71ecc0bf10a1483479852956ef87aa69686: Status 404 returned error can't find the container with id c19ac35cd402974dae9fb0b6b6a4f71ecc0bf10a1483479852956ef87aa69686
Jul 29 10:50:42.383208 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:42.383178    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerStarted","Data":"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"}
Jul 29 10:50:42.383396 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:50:42.383241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerStarted","Data":"c19ac35cd402974dae9fb0b6b6a4f71ecc0bf10a1483479852956ef87aa69686"}
Jul 29 10:51:00.463994 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:00.463963    2551 generic.go:356] "Generic (PLEG): container finished" podID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerID="d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a" exitCode=0
Jul 29 10:51:00.464332 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:00.464039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerDied","Data":"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"}
Jul 29 10:51:01.471033 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:01.470998    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerStarted","Data":"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"}
Jul 29 10:51:01.471385 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:01.471065    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerStarted","Data":"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"}
Jul 29 10:51:01.471385 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:01.471263    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:51:01.492514 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:01.492461    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" podStartSLOduration=2.12900536 podStartE2EDuration="20.492445062s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.363439702s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:01.490529295 +0000 UTC m=+1313.392757694" watchObservedRunningTime="2026-07-29 10:51:01.492445062 +0000 UTC m=+1313.394673462"
Jul 29 10:51:02.475616 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:02.475586    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:51:08.485550 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:51:08.485524    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:52:38.729871 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:38.729840    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:52:48.050871 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.050840    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:52:48.051345 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.051085    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kserve-container" containerID="cri-o://02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b" gracePeriod=30
Jul 29 10:52:48.051345 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.051102    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kube-rbac-proxy" containerID="cri-o://29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc" gracePeriod=30
Jul 29 10:52:48.481926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.481881    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 29 10:52:48.726446 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.726405    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 29 10:52:48.952546 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.952504    2551 generic.go:356] "Generic (PLEG): container finished" podID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerID="29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc" exitCode=2
Jul 29 10:52:48.952725 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:48.952549    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerDied","Data":"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"}
Jul 29 10:52:49.302512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.302491    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:52:49.413320 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413264    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls\") pod \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") "
Jul 29 10:52:49.413414 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413333    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert\") pod \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") "
Jul 29 10:52:49.413414 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413398    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location\") pod \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") "
Jul 29 10:52:49.413494 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413424    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghcvf\" (UniqueName: \"kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf\") pod \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") "
Jul 29 10:52:49.413494 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413446    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config\") pod \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\" (UID: \"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975\") "
Jul 29 10:52:49.413741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413710    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" (UID: "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:52:49.413902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.413875    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" (UID: "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:52:49.415329 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.415308    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" (UID: "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:52:49.415417 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.415394    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf" (OuterVolumeSpecName: "kube-api-access-ghcvf") pod "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" (UID: "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975"). InnerVolumeSpecName "kube-api-access-ghcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:52:49.418200 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.418176    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" (UID: "34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:52:49.514835 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.514812    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:52:49.514835 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.514834    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghcvf\" (UniqueName: \"kubernetes.io/projected/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-api-access-ghcvf\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:52:49.514946 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.514843    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:52:49.514946 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.514854    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:52:49.514946 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.514863    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:52:49.960293 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.960263    2551 generic.go:356] "Generic (PLEG): container finished" podID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerID="02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b" exitCode=0
Jul 29 10:52:49.960434 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.960334    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"
Jul 29 10:52:49.960434 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.960359    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerDied","Data":"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"}
Jul 29 10:52:49.960525 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.960437    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn" event={"ID":"34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975","Type":"ContainerDied","Data":"c19ac35cd402974dae9fb0b6b6a4f71ecc0bf10a1483479852956ef87aa69686"}
Jul 29 10:52:49.960525 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.960473    2551 scope.go:117] "RemoveContainer" containerID="29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"
Jul 29 10:52:49.970366 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.970349    2551 scope.go:117] "RemoveContainer" containerID="02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"
Jul 29 10:52:49.978492 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.978464    2551 scope.go:117] "RemoveContainer" containerID="d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"
Jul 29 10:52:49.983220 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.983196    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:52:49.986769 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.986743    2551 scope.go:117] "RemoveContainer" containerID="29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"
Jul 29 10:52:49.986885 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.986755    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7zqxn"]
Jul 29 10:52:49.987145 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:52:49.987106    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc\": container with ID starting with 29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc not found: ID does not exist" containerID="29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"
Jul 29 10:52:49.987227 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.987152    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc"} err="failed to get container status \"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc\": rpc error: code = NotFound desc = could not find container \"29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc\": container with ID starting with 29d8f3cfab07ced760405108df838930d1942a6290af899b8db7c96b6a41a7bc not found: ID does not exist"
Jul 29 10:52:49.987227 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.987176    2551 scope.go:117] "RemoveContainer" containerID="02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"
Jul 29 10:52:49.987500 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:52:49.987481    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b\": container with ID starting with 02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b not found: ID does not exist" containerID="02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"
Jul 29 10:52:49.987539 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.987507    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b"} err="failed to get container status \"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b\": rpc error: code = NotFound desc = could not find container \"02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b\": container with ID starting with 02577a8e0e3ccedcf3e70247e87e0218ce13032e5a9d8b5ba4966c0e04aca05b not found: ID does not exist"
Jul 29 10:52:49.987539 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.987526    2551 scope.go:117] "RemoveContainer" containerID="d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"
Jul 29 10:52:49.987774 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:52:49.987755    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a\": container with ID starting with d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a not found: ID does not exist" containerID="d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"
Jul 29 10:52:49.987818 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:49.987782    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a"} err="failed to get container status \"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a\": rpc error: code = NotFound desc = could not find container \"d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a\": container with ID starting with d64f413be33fa14e251f0db08c234b25178f8057873728011292649af4524c9a not found: ID does not exist"
Jul 29 10:52:50.727642 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:50.727608    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" path="/var/lib/kubelet/pods/34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975/volumes"
Jul 29 10:52:53.180371 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180341    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180683    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kserve-container"
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180695    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kserve-container"
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180702    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kube-rbac-proxy"
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180708    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kube-rbac-proxy"
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180725    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="storage-initializer"
Jul 29 10:52:53.180751 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180731    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="storage-initializer"
Jul 29 10:52:53.180937 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180784    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kube-rbac-proxy"
Jul 29 10:52:53.180937 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.180792    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="34760b6f-bbfa-4dfc-a2eb-af6ce3fdc975" containerName="kserve-container"
Jul 29 10:52:53.185578 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.185562    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.188287 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188246    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 29 10:52:53.188287 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188265    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:52:53.188512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188292    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:52:53.188512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188308    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:52:53.188512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188265    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:52:53.188512 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.188273    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 29 10:52:53.189240 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.189225    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:52:53.194162 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.194138    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:52:53.242758 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.242738    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.242861 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.242769    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjl9\" (UniqueName: \"kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.242861 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.242794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.242980 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.242890    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.242980 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.242925    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344059 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344083    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344104    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344167    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjl9\" (UniqueName: \"kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344554 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344741 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344724    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.344855 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.344836    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.346587 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.346566    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.355354 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.355331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjl9\" (UniqueName: \"kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.496342 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.496316    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:52:53.627979 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.627951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:52:53.630411 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:52:53.630384    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a4079_d16d_4f38_9dc4_3b06f6987b53.slice/crio-9097033f706f7652e1d5411e39c1787164367d5fb461c0d073f75f4d92f3fbd8 WatchSource:0}: Error finding container 9097033f706f7652e1d5411e39c1787164367d5fb461c0d073f75f4d92f3fbd8: Status 404 returned error can't find the container with id 9097033f706f7652e1d5411e39c1787164367d5fb461c0d073f75f4d92f3fbd8
Jul 29 10:52:53.981266 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.981229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerStarted","Data":"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"}
Jul 29 10:52:53.981418 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:52:53.981292    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerStarted","Data":"9097033f706f7652e1d5411e39c1787164367d5fb461c0d073f75f4d92f3fbd8"}
Jul 29 10:53:24.110574 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:24.110539    2551 generic.go:356] "Generic (PLEG): container finished" podID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerID="2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa" exitCode=0
Jul 29 10:53:24.111181 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:24.110623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerDied","Data":"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"}
Jul 29 10:53:25.117529 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:25.117493    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerStarted","Data":"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"}
Jul 29 10:53:25.117904 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:25.117563    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerStarted","Data":"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"}
Jul 29 10:53:25.117904 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:25.117749    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:53:25.138048 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:25.137997    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" podStartSLOduration=2.393962154 podStartE2EDuration="32.137979474s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.74401732s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:53:25.135800611 +0000 UTC m=+1457.038029012" watchObservedRunningTime="2026-07-29 10:53:25.137979474 +0000 UTC m=+1457.040207873"
Jul 29 10:53:26.123431 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:26.123398    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:53:32.134140 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:53:32.134101    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:54:08.657501 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:54:08.657468    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:54:08.659898 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:54:08.659880    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:55:01.727226 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:01.727198    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:55:09.293312 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:09.293281    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:55:09.293790 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:09.293512    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kserve-container" containerID="cri-o://b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84" gracePeriod=30
Jul 29 10:55:09.293790 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:09.293549    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kube-rbac-proxy" containerID="cri-o://d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538" gracePeriod=30
Jul 29 10:55:09.562179 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:09.562069    2551 generic.go:356] "Generic (PLEG): container finished" podID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerID="d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538" exitCode=2
Jul 29 10:55:09.562179 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:09.562108    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerDied","Data":"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"}
Jul 29 10:55:10.538615 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.538588    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:55:10.568517 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.568487    2551 generic.go:356] "Generic (PLEG): container finished" podID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerID="b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84" exitCode=0
Jul 29 10:55:10.568648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.568534    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"
Jul 29 10:55:10.568648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.568561    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerDied","Data":"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"}
Jul 29 10:55:10.568648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.568614    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2" event={"ID":"175a4079-d16d-4f38-9dc4-3b06f6987b53","Type":"ContainerDied","Data":"9097033f706f7652e1d5411e39c1787164367d5fb461c0d073f75f4d92f3fbd8"}
Jul 29 10:55:10.568648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.568641    2551 scope.go:117] "RemoveContainer" containerID="d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"
Jul 29 10:55:10.578115 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578096    2551 scope.go:117] "RemoveContainer" containerID="b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"
Jul 29 10:55:10.578253 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578235    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert\") pod \"175a4079-d16d-4f38-9dc4-3b06f6987b53\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") "
Jul 29 10:55:10.578389 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578371    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config\") pod \"175a4079-d16d-4f38-9dc4-3b06f6987b53\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") "
Jul 29 10:55:10.578474 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578460    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location\") pod \"175a4079-d16d-4f38-9dc4-3b06f6987b53\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") "
Jul 29 10:55:10.578540 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578500    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tjl9\" (UniqueName: \"kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9\") pod \"175a4079-d16d-4f38-9dc4-3b06f6987b53\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") "
Jul 29 10:55:10.578540 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578533    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls\") pod \"175a4079-d16d-4f38-9dc4-3b06f6987b53\" (UID: \"175a4079-d16d-4f38-9dc4-3b06f6987b53\") "
Jul 29 10:55:10.578721 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578698    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "175a4079-d16d-4f38-9dc4-3b06f6987b53" (UID: "175a4079-d16d-4f38-9dc4-3b06f6987b53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:55:10.578814 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578796    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:10.578941 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.578916    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "175a4079-d16d-4f38-9dc4-3b06f6987b53" (UID: "175a4079-d16d-4f38-9dc4-3b06f6987b53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:55:10.580708 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.580678    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "175a4079-d16d-4f38-9dc4-3b06f6987b53" (UID: "175a4079-d16d-4f38-9dc4-3b06f6987b53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:55:10.580910 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.580885    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9" (OuterVolumeSpecName: "kube-api-access-8tjl9") pod "175a4079-d16d-4f38-9dc4-3b06f6987b53" (UID: "175a4079-d16d-4f38-9dc4-3b06f6987b53"). InnerVolumeSpecName "kube-api-access-8tjl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:55:10.583651 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.583630    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "175a4079-d16d-4f38-9dc4-3b06f6987b53" (UID: "175a4079-d16d-4f38-9dc4-3b06f6987b53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:10.594710 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.594684    2551 scope.go:117] "RemoveContainer" containerID="2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"
Jul 29 10:55:10.601571 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.601548    2551 scope.go:117] "RemoveContainer" containerID="d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"
Jul 29 10:55:10.601828 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:10.601807    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538\": container with ID starting with d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538 not found: ID does not exist" containerID="d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"
Jul 29 10:55:10.601902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.601841    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538"} err="failed to get container status \"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538\": rpc error: code = NotFound desc = could not find container \"d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538\": container with ID starting with d623a126c718fae73426272ce7323cb845b96be9a1545dcd0aacae9579d40538 not found: ID does not exist"
Jul 29 10:55:10.601902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.601867    2551 scope.go:117] "RemoveContainer" containerID="b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"
Jul 29 10:55:10.602117 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:10.602098    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84\": container with ID starting with b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84 not found: ID does not exist" containerID="b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"
Jul 29 10:55:10.602182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.602123    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84"} err="failed to get container status \"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84\": rpc error: code = NotFound desc = could not find container \"b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84\": container with ID starting with b695e1fe3989637710871b3abdd3c9a6eef625560b257e967d3d470a269bef84 not found: ID does not exist"
Jul 29 10:55:10.602182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.602153    2551 scope.go:117] "RemoveContainer" containerID="2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"
Jul 29 10:55:10.602345 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:10.602329    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa\": container with ID starting with 2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa not found: ID does not exist" containerID="2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"
Jul 29 10:55:10.602384 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.602350    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa"} err="failed to get container status \"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa\": rpc error: code = NotFound desc = could not find container \"2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa\": container with ID starting with 2a5819960a8b62f1c2cecc61c0f6b148fcd210a86398bc65d5134083963d6afa not found: ID does not exist"
Jul 29 10:55:10.679469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.679421    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/175a4079-d16d-4f38-9dc4-3b06f6987b53-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:10.679469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.679441    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjl9\" (UniqueName: \"kubernetes.io/projected/175a4079-d16d-4f38-9dc4-3b06f6987b53-kube-api-access-8tjl9\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:10.679469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.679450    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175a4079-d16d-4f38-9dc4-3b06f6987b53-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:10.679469 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.679460    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/175a4079-d16d-4f38-9dc4-3b06f6987b53-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:10.884546 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.884528    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:55:10.888084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:10.888063    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-9wrf2"]
Jul 29 10:55:12.727714 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:12.727682    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" path="/var/lib/kubelet/pods/175a4079-d16d-4f38-9dc4-3b06f6987b53/volumes"
Jul 29 10:55:14.434285 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434250    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434578    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="storage-initializer"
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434589    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="storage-initializer"
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434601    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kserve-container"
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434607    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kserve-container"
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434618    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kube-rbac-proxy"
Jul 29 10:55:14.434665 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434624    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kube-rbac-proxy"
Jul 29 10:55:14.434885 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434689    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kserve-container"
Jul 29 10:55:14.434885 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.434701    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="175a4079-d16d-4f38-9dc4-3b06f6987b53" containerName="kube-rbac-proxy"
Jul 29 10:55:14.439507 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.439491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.441702 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.441680    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 29 10:55:14.441810 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.441720    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 29 10:55:14.441951 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.441937    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:55:14.442019 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.441975    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:55:14.442189 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.442174    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:55:14.443176 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.443160    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:55:14.443268 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.443189    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:55:14.446784 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.446763    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:14.509649 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.509625    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zbz\" (UniqueName: \"kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.509747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.509658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.509747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.509683    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.509747 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.509710    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.509873 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.509771    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.610897 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.610875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zbz\" (UniqueName: \"kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611003 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.610908    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611003 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.610933    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611003 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.610958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611003 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.610987    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611384 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.611361    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.611603    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.611672 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.611630    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.613380 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.613364    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.618538 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.618516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zbz\" (UniqueName: \"kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.751153 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.751117    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:14.878310 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:14.878282    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:14.880205 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:55:14.880179    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afa640c_1ae4_4c44_8a83_2883636acba1.slice/crio-ccdb9cea01c25bad637a2db6fbd73433244692fc3878d7dae9faf8d17ca62fe2 WatchSource:0}: Error finding container ccdb9cea01c25bad637a2db6fbd73433244692fc3878d7dae9faf8d17ca62fe2: Status 404 returned error can't find the container with id ccdb9cea01c25bad637a2db6fbd73433244692fc3878d7dae9faf8d17ca62fe2
Jul 29 10:55:15.594599 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:15.594569    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerStarted","Data":"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"}
Jul 29 10:55:15.595050 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:15.594634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerStarted","Data":"ccdb9cea01c25bad637a2db6fbd73433244692fc3878d7dae9faf8d17ca62fe2"}
Jul 29 10:55:22.624410 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:22.624380    2551 generic.go:356] "Generic (PLEG): container finished" podID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerID="e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00" exitCode=0
Jul 29 10:55:22.624728 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:22.624408    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerDied","Data":"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"}
Jul 29 10:55:23.631355 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.631322    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerStarted","Data":"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"}
Jul 29 10:55:23.631742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.631383    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerStarted","Data":"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"}
Jul 29 10:55:23.631742 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.631682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:23.631861 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.631842    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:23.633057 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.633034    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 29 10:55:23.651403 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:23.651359    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podStartSLOduration=2.052700219 podStartE2EDuration="9.651346919s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.5986467s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:55:23.649176484 +0000 UTC m=+1575.551404885" watchObservedRunningTime="2026-07-29 10:55:23.651346919 +0000 UTC m=+1575.553575316"
Jul 29 10:55:24.636599 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:24.636548    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 29 10:55:25.640746 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:25.640706    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 29 10:55:30.645234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:30.645194    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:30.645724 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:30.645701    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 29 10:55:40.646297 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:40.646267    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:56.514370 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:56.514331    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:56.515036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:56.514582    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container" containerID="cri-o://2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f" gracePeriod=30
Jul 29 10:55:56.515036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:56.514606    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kube-rbac-proxy" containerID="cri-o://dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76" gracePeriod=30
Jul 29 10:55:56.783468 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:56.783383    2551 generic.go:356] "Generic (PLEG): container finished" podID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerID="dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76" exitCode=2
Jul 29 10:55:56.783468 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:56.783423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerDied","Data":"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"}
Jul 29 10:55:58.159078 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.159048    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:58.289727 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.289698    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8zbz\" (UniqueName: \"kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz\") pod \"6afa640c-1ae4-4c44-8a83-2883636acba1\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") "
Jul 29 10:55:58.289921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.289751    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config\") pod \"6afa640c-1ae4-4c44-8a83-2883636acba1\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") "
Jul 29 10:55:58.289921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.289779    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls\") pod \"6afa640c-1ae4-4c44-8a83-2883636acba1\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") "
Jul 29 10:55:58.289921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.289798    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert\") pod \"6afa640c-1ae4-4c44-8a83-2883636acba1\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") "
Jul 29 10:55:58.289921 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.289836    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location\") pod \"6afa640c-1ae4-4c44-8a83-2883636acba1\" (UID: \"6afa640c-1ae4-4c44-8a83-2883636acba1\") "
Jul 29 10:55:58.290273 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.290240    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6afa640c-1ae4-4c44-8a83-2883636acba1" (UID: "6afa640c-1ae4-4c44-8a83-2883636acba1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:55:58.290360 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.290255    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6afa640c-1ae4-4c44-8a83-2883636acba1" (UID: "6afa640c-1ae4-4c44-8a83-2883636acba1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:55:58.290854 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.290828    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6afa640c-1ae4-4c44-8a83-2883636acba1" (UID: "6afa640c-1ae4-4c44-8a83-2883636acba1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:58.291934 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.291905    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz" (OuterVolumeSpecName: "kube-api-access-z8zbz") pod "6afa640c-1ae4-4c44-8a83-2883636acba1" (UID: "6afa640c-1ae4-4c44-8a83-2883636acba1"). InnerVolumeSpecName "kube-api-access-z8zbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:55:58.291934 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.291913    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6afa640c-1ae4-4c44-8a83-2883636acba1" (UID: "6afa640c-1ae4-4c44-8a83-2883636acba1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:55:58.390445 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.390407    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6afa640c-1ae4-4c44-8a83-2883636acba1-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:58.390445 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.390443    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:58.390445 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.390453    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6afa640c-1ae4-4c44-8a83-2883636acba1-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:58.390651 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.390464    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zbz\" (UniqueName: \"kubernetes.io/projected/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-api-access-z8zbz\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:58.390651 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.390474    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6afa640c-1ae4-4c44-8a83-2883636acba1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:55:58.794605 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.794572    2551 generic.go:356] "Generic (PLEG): container finished" podID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerID="2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f" exitCode=0
Jul 29 10:55:58.794773 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.794640    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"
Jul 29 10:55:58.794773 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.794652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerDied","Data":"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"}
Jul 29 10:55:58.794773 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.794722    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf" event={"ID":"6afa640c-1ae4-4c44-8a83-2883636acba1","Type":"ContainerDied","Data":"ccdb9cea01c25bad637a2db6fbd73433244692fc3878d7dae9faf8d17ca62fe2"}
Jul 29 10:55:58.794773 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.794758    2551 scope.go:117] "RemoveContainer" containerID="dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"
Jul 29 10:55:58.802876 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.802844    2551 scope.go:117] "RemoveContainer" containerID="2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"
Jul 29 10:55:58.810173 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.810153    2551 scope.go:117] "RemoveContainer" containerID="e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"
Jul 29 10:55:58.811990 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.811973    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:58.815436 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.815416    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-z25nf"]
Jul 29 10:55:58.817847 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.817835    2551 scope.go:117] "RemoveContainer" containerID="dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"
Jul 29 10:55:58.818086 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:58.818066    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76\": container with ID starting with dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76 not found: ID does not exist" containerID="dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"
Jul 29 10:55:58.818157 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.818093    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76"} err="failed to get container status \"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76\": rpc error: code = NotFound desc = could not find container \"dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76\": container with ID starting with dffdcb420a1881d4ca5851c4500d7207085f7009a0af628cf22674152c98fc76 not found: ID does not exist"
Jul 29 10:55:58.818157 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.818111    2551 scope.go:117] "RemoveContainer" containerID="2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"
Jul 29 10:55:58.818364 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:58.818349    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f\": container with ID starting with 2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f not found: ID does not exist" containerID="2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"
Jul 29 10:55:58.818403 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.818370    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f"} err="failed to get container status \"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f\": rpc error: code = NotFound desc = could not find container \"2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f\": container with ID starting with 2b351fd50b1bb5e87786388ecad80b29bc37223a52ac274b88e00b2f6bfcde9f not found: ID does not exist"
Jul 29 10:55:58.818403 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.818388    2551 scope.go:117] "RemoveContainer" containerID="e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"
Jul 29 10:55:58.818622 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:55:58.818603    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00\": container with ID starting with e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00 not found: ID does not exist" containerID="e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"
Jul 29 10:55:58.818661 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:55:58.818629    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00"} err="failed to get container status \"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00\": rpc error: code = NotFound desc = could not find container \"e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00\": container with ID starting with e72fe71470915796e7abd28eefef5859276a4d77c761022dbdff6344234c1b00 not found: ID does not exist"
Jul 29 10:56:00.727999 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:00.727927    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" path="/var/lib/kubelet/pods/6afa640c-1ae4-4c44-8a83-2883636acba1/volumes"
Jul 29 10:56:01.685231 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685199    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:01.685518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685504    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kube-rbac-proxy"
Jul 29 10:56:01.685558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685520    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kube-rbac-proxy"
Jul 29 10:56:01.685558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685533    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="storage-initializer"
Jul 29 10:56:01.685558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685538    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="storage-initializer"
Jul 29 10:56:01.685558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685552    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container"
Jul 29 10:56:01.685686 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685560    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container"
Jul 29 10:56:01.685686 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685616    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kube-rbac-proxy"
Jul 29 10:56:01.685686 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.685628    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afa640c-1ae4-4c44-8a83-2883636acba1" containerName="kserve-container"
Jul 29 10:56:01.690026 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.690009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.692562 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692527    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:56:01.692562 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692550    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:56:01.692810 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692687    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 29 10:56:01.692810 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692771    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:56:01.692943 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692923    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 29 10:56:01.693013 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.692948    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:56:01.693453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.693439    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:56:01.698413 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.698392    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:01.817884 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.817854    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.818374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.817889    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.818374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.817910    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.818374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.817940    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.818374 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.817998    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276l6\" (UniqueName: \"kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919251 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919219    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276l6\" (UniqueName: \"kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919339    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.919814 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.919791    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.920083 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.920062    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.920160 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.920116    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.921970 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.921949    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:01.928926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:01.928903    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-276l6\" (UniqueName: \"kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:02.001297 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:02.001276    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:02.131464 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:02.131439    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:02.133882 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:56:02.133857    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae457223_55f0_463a_a544_2d2cd32ce7ea.slice/crio-cbd985af88828912236fe3172e7d0896eded513fc1d0850d564fe43dbe0d5fb9 WatchSource:0}: Error finding container cbd985af88828912236fe3172e7d0896eded513fc1d0850d564fe43dbe0d5fb9: Status 404 returned error can't find the container with id cbd985af88828912236fe3172e7d0896eded513fc1d0850d564fe43dbe0d5fb9
Jul 29 10:56:02.814282 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:02.814245    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerStarted","Data":"31c8151cb302a63031c2190fefa1b39161e1373af344ffc0e946f8fa3d97cd81"}
Jul 29 10:56:02.814460 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:02.814302    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerStarted","Data":"cbd985af88828912236fe3172e7d0896eded513fc1d0850d564fe43dbe0d5fb9"}
Jul 29 10:56:06.832935 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:06.832898    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerID="31c8151cb302a63031c2190fefa1b39161e1373af344ffc0e946f8fa3d97cd81" exitCode=0
Jul 29 10:56:06.833376 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:06.832970    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerDied","Data":"31c8151cb302a63031c2190fefa1b39161e1373af344ffc0e946f8fa3d97cd81"}
Jul 29 10:56:07.840546 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:07.840506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerStarted","Data":"2bc1188063231630e95024f0df3a3d8538fb33a459e14a2bda061671e5199d27"}
Jul 29 10:56:07.840926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:07.840589    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerStarted","Data":"77ff7eb40ea95b73093174fc5057c51740ae29bdeb4a260d2bc8529835032a52"}
Jul 29 10:56:07.840926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:07.840824    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:07.861924 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:07.861866    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podStartSLOduration=2.674914222 podStartE2EDuration="6.861846724s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.186932502s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:07.860636359 +0000 UTC m=+1619.762864757" watchObservedRunningTime="2026-07-29 10:56:07.861846724 +0000 UTC m=+1619.764075123"
Jul 29 10:56:08.844806 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:08.844764    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:08.845978 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:08.845947    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 29 10:56:09.849494 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:09.849453    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 29 10:56:14.853726 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:14.853695    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:14.854299 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:14.854274    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 29 10:56:24.855053 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:24.855022    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:36.773083 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:36.773051    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:36.773613 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:36.773303    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container" containerID="cri-o://77ff7eb40ea95b73093174fc5057c51740ae29bdeb4a260d2bc8529835032a52" gracePeriod=30
Jul 29 10:56:36.773613 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:36.773329    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kube-rbac-proxy" containerID="cri-o://2bc1188063231630e95024f0df3a3d8538fb33a459e14a2bda061671e5199d27" gracePeriod=30
Jul 29 10:56:36.970855 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:36.970825    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerID="2bc1188063231630e95024f0df3a3d8538fb33a459e14a2bda061671e5199d27" exitCode=2
Jul 29 10:56:36.971002 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:36.970895    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerDied","Data":"2bc1188063231630e95024f0df3a3d8538fb33a459e14a2bda061671e5199d27"}
Jul 29 10:56:37.977769 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:37.977742    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerID="77ff7eb40ea95b73093174fc5057c51740ae29bdeb4a260d2bc8529835032a52" exitCode=0
Jul 29 10:56:37.978117 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:37.977813    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerDied","Data":"77ff7eb40ea95b73093174fc5057c51740ae29bdeb4a260d2bc8529835032a52"}
Jul 29 10:56:38.011924 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.011900    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:38.122617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.122551    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls\") pod \"ae457223-55f0-463a-a544-2d2cd32ce7ea\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") "
Jul 29 10:56:38.122617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.122599    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config\") pod \"ae457223-55f0-463a-a544-2d2cd32ce7ea\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") "
Jul 29 10:56:38.122802 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.122638    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location\") pod \"ae457223-55f0-463a-a544-2d2cd32ce7ea\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") "
Jul 29 10:56:38.122802 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.122671    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert\") pod \"ae457223-55f0-463a-a544-2d2cd32ce7ea\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") "
Jul 29 10:56:38.122802 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.122729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-276l6\" (UniqueName: \"kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6\") pod \"ae457223-55f0-463a-a544-2d2cd32ce7ea\" (UID: \"ae457223-55f0-463a-a544-2d2cd32ce7ea\") "
Jul 29 10:56:38.123036 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.123012    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae457223-55f0-463a-a544-2d2cd32ce7ea" (UID: "ae457223-55f0-463a-a544-2d2cd32ce7ea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:56:38.123117 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.123090    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae457223-55f0-463a-a544-2d2cd32ce7ea" (UID: "ae457223-55f0-463a-a544-2d2cd32ce7ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:56:38.123656 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.123631    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae457223-55f0-463a-a544-2d2cd32ce7ea" (UID: "ae457223-55f0-463a-a544-2d2cd32ce7ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:38.124581 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.124562    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae457223-55f0-463a-a544-2d2cd32ce7ea" (UID: "ae457223-55f0-463a-a544-2d2cd32ce7ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:56:38.124666 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.124652    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6" (OuterVolumeSpecName: "kube-api-access-276l6") pod "ae457223-55f0-463a-a544-2d2cd32ce7ea" (UID: "ae457223-55f0-463a-a544-2d2cd32ce7ea"). InnerVolumeSpecName "kube-api-access-276l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:56:38.223848 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.223825    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae457223-55f0-463a-a544-2d2cd32ce7ea-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:56:38.223848 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.223847    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:56:38.223972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.223857    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276l6\" (UniqueName: \"kubernetes.io/projected/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-api-access-276l6\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:56:38.223972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.223866    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae457223-55f0-463a-a544-2d2cd32ce7ea-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:56:38.223972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.223876    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae457223-55f0-463a-a544-2d2cd32ce7ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:56:38.983631 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.983607    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"
Jul 29 10:56:38.983993 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.983604    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx" event={"ID":"ae457223-55f0-463a-a544-2d2cd32ce7ea","Type":"ContainerDied","Data":"cbd985af88828912236fe3172e7d0896eded513fc1d0850d564fe43dbe0d5fb9"}
Jul 29 10:56:38.983993 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.983742    2551 scope.go:117] "RemoveContainer" containerID="2bc1188063231630e95024f0df3a3d8538fb33a459e14a2bda061671e5199d27"
Jul 29 10:56:38.991823 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.991810    2551 scope.go:117] "RemoveContainer" containerID="77ff7eb40ea95b73093174fc5057c51740ae29bdeb4a260d2bc8529835032a52"
Jul 29 10:56:38.999269 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:38.999250    2551 scope.go:117] "RemoveContainer" containerID="31c8151cb302a63031c2190fefa1b39161e1373af344ffc0e946f8fa3d97cd81"
Jul 29 10:56:39.002621 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:39.002594    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:39.008791 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:39.008774    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-bgwtx"]
Jul 29 10:56:40.727924 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:40.727882    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" path="/var/lib/kubelet/pods/ae457223-55f0-463a-a544-2d2cd32ce7ea/volumes"
Jul 29 10:56:41.949883 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.949853    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:56:41.950405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950385    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="storage-initializer"
Jul 29 10:56:41.950486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950411    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="storage-initializer"
Jul 29 10:56:41.950486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950424    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kube-rbac-proxy"
Jul 29 10:56:41.950486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950433    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kube-rbac-proxy"
Jul 29 10:56:41.950486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950471    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container"
Jul 29 10:56:41.950486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950480    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container"
Jul 29 10:56:41.950737 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950547    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kserve-container"
Jul 29 10:56:41.950737 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.950561    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae457223-55f0-463a-a544-2d2cd32ce7ea" containerName="kube-rbac-proxy"
Jul 29 10:56:41.953908 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.953889    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:41.956462 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.956443    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 29 10:56:41.956566 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.956544    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:56:41.956851 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.956816    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:56:41.957507 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.957486    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:56:41.957605 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.957562    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 29 10:56:41.957674 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.957615    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:56:41.957674 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.957615    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:56:41.964168 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.964148    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:56:41.995633 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.995615    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:56:41.999175 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:41.999160    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.001623 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.001604    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 29 10:56:42.009330 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.009310    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:56:42.051491 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.051589 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.051589 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsjz\" (UniqueName: \"kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.051589 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051532    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.051692 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051608    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhxc\" (UniqueName: \"kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.051692 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051649    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.051692 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051683    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.051788 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051727    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.051788 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.051788 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.051764    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152239 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152210    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.152347 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.152347 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152274    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152347 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152308    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.152347 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152324    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152569 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152382    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsjz\" (UniqueName: \"kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152569 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152435    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152569 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152464    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhxc\" (UniqueName: \"kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.152569 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.152569 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152530    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.152884 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.152700    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.153032 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.153010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.153123 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.153060    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.153123 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.153061    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.153259 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.153242    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.153495 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.153473    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.154812 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.154795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.154877 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.154861    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.160104 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.160082    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsjz\" (UniqueName: \"kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz\") pod \"isvc-canary-create-predictor-5d59fd866b-jxtpx\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.160317 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.160300    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhxc\" (UniqueName: \"kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc\") pod \"isvc-canary-create-v2-predictor-75f789cb99-5npt5\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.263722 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.263659    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:42.308478 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.308452    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:42.403706 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.403676    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:56:42.405648 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:56:42.405617    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1321a3_8c2a_4904_aa06_96e6bca7b60f.slice/crio-21c4db8bde667f125a13a30f62be41516b9082a67631bd79df8b05b28283c165 WatchSource:0}: Error finding container 21c4db8bde667f125a13a30f62be41516b9082a67631bd79df8b05b28283c165: Status 404 returned error can't find the container with id 21c4db8bde667f125a13a30f62be41516b9082a67631bd79df8b05b28283c165
Jul 29 10:56:42.447488 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:42.447444    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:56:42.450903 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:56:42.450880    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9e6bf2_56b1_4a81_a076_eaa8cdef97d7.slice/crio-54a7d872084fc46ead5ab47a55f806a04c640eeaa9257ba5b1879f32652d5dc6 WatchSource:0}: Error finding container 54a7d872084fc46ead5ab47a55f806a04c640eeaa9257ba5b1879f32652d5dc6: Status 404 returned error can't find the container with id 54a7d872084fc46ead5ab47a55f806a04c640eeaa9257ba5b1879f32652d5dc6
Jul 29 10:56:43.005286 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:43.005244    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerStarted","Data":"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"}
Jul 29 10:56:43.005735 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:43.005331    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerStarted","Data":"54a7d872084fc46ead5ab47a55f806a04c640eeaa9257ba5b1879f32652d5dc6"}
Jul 29 10:56:43.006859 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:43.006833    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerStarted","Data":"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"}
Jul 29 10:56:43.007029 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:43.006881    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerStarted","Data":"21c4db8bde667f125a13a30f62be41516b9082a67631bd79df8b05b28283c165"}
Jul 29 10:56:47.030600 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:47.030559    2551 generic.go:356] "Generic (PLEG): container finished" podID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerID="f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4" exitCode=0
Jul 29 10:56:47.031046 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:47.030629    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerDied","Data":"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"}
Jul 29 10:56:47.032242 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:47.032221    2551 generic.go:356] "Generic (PLEG): container finished" podID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerID="eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1" exitCode=0
Jul 29 10:56:47.032318 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:47.032299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerDied","Data":"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"}
Jul 29 10:56:48.039504 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.039468    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerStarted","Data":"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"}
Jul 29 10:56:48.039974 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.039538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerStarted","Data":"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"}
Jul 29 10:56:48.039974 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.039807    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:48.039974 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.039965    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:48.041233 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.041208    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:56:48.041797 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.041779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerStarted","Data":"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"}
Jul 29 10:56:48.041887 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.041822    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerStarted","Data":"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"}
Jul 29 10:56:48.042005 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.041990    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:48.042072 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.042035    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:48.043174 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.043150    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:56:48.061484 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.061443    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podStartSLOduration=3.210807394 podStartE2EDuration="7.061429451s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.850622057s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:48.060150395 +0000 UTC m=+1659.962378790" watchObservedRunningTime="2026-07-29 10:56:48.061429451 +0000 UTC m=+1659.963657848"
Jul 29 10:56:48.081745 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:48.081703    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podStartSLOduration=3.265400252 podStartE2EDuration="7.081689722s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81628947s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:48.08094814 +0000 UTC m=+1659.983176537" watchObservedRunningTime="2026-07-29 10:56:48.081689722 +0000 UTC m=+1659.983918183"
Jul 29 10:56:49.046219 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:49.046179    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:56:49.046666 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:49.046290    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:56:54.050833 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:54.050806    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:56:54.051276 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:54.051202    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:56:54.051276 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:54.051228    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:56:54.051671 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:56:54.051641    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:04.051925 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:04.051878    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:57:04.052353 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:04.051878    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:14.052045 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:14.052003    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:57:14.052045 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:14.052003    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:24.052195 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:24.052154    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:24.052195 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:24.052171    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:57:34.051670 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:34.051585    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:34.052030 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:34.051586    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:57:44.051974 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:44.051937    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:44.052431 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:44.051942    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:57:54.051904 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:54.051863    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:57:54.052394 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:57:54.051864    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:58:04.052712 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:04.052683    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:58:04.053173 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:04.052762    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:58:12.067787 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.067758    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:58:12.068363 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.068042    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" containerID="cri-o://8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3" gracePeriod=30
Jul 29 10:58:12.068363 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.068097    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kube-rbac-proxy" containerID="cri-o://f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9" gracePeriod=30
Jul 29 10:58:12.073124 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.073091    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:58:12.073410 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.073361    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" containerID="cri-o://fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd" gracePeriod=30
Jul 29 10:58:12.073551 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.073413    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kube-rbac-proxy" containerID="cri-o://d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0" gracePeriod=30
Jul 29 10:58:12.401158 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.401051    2551 generic.go:356] "Generic (PLEG): container finished" podID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerID="d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0" exitCode=2
Jul 29 10:58:12.401319 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.401145    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerDied","Data":"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"}
Jul 29 10:58:12.403010 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.402990    2551 generic.go:356] "Generic (PLEG): container finished" podID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerID="f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9" exitCode=2
Jul 29 10:58:12.403151 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:12.403038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerDied","Data":"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"}
Jul 29 10:58:14.047121 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:14.047090    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 29 10:58:14.047121 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:14.047083    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 29 10:58:14.051458 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:14.051438    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 29 10:58:14.051583 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:14.051562    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 29 10:58:16.349293 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.349267    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:58:16.352041 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.352022    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:58:16.422432 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.422401    2551 generic.go:356] "Generic (PLEG): container finished" podID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerID="fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd" exitCode=0
Jul 29 10:58:16.422588 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.422463    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"
Jul 29 10:58:16.422588 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.422483    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerDied","Data":"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"}
Jul 29 10:58:16.422588 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.422553    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5" event={"ID":"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7","Type":"ContainerDied","Data":"54a7d872084fc46ead5ab47a55f806a04c640eeaa9257ba5b1879f32652d5dc6"}
Jul 29 10:58:16.422749 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.422595    2551 scope.go:117] "RemoveContainer" containerID="d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"
Jul 29 10:58:16.424364 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.424343    2551 generic.go:356] "Generic (PLEG): container finished" podID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerID="8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3" exitCode=0
Jul 29 10:58:16.424457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.424375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerDied","Data":"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"}
Jul 29 10:58:16.424457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.424391    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"
Jul 29 10:58:16.424457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.424421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx" event={"ID":"cd1321a3-8c2a-4904-aa06-96e6bca7b60f","Type":"ContainerDied","Data":"21c4db8bde667f125a13a30f62be41516b9082a67631bd79df8b05b28283c165"}
Jul 29 10:58:16.433225 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.432453    2551 scope.go:117] "RemoveContainer" containerID="fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"
Jul 29 10:58:16.440518 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.440500    2551 scope.go:117] "RemoveContainer" containerID="f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"
Jul 29 10:58:16.447772 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.447757    2551 scope.go:117] "RemoveContainer" containerID="d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"
Jul 29 10:58:16.448013 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.447997    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0\": container with ID starting with d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0 not found: ID does not exist" containerID="d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"
Jul 29 10:58:16.448064 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448022    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0"} err="failed to get container status \"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0\": rpc error: code = NotFound desc = could not find container \"d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0\": container with ID starting with d7e987193496dcc8a582a0c273442f7a5c5caee4a25d724192f5f45522a399e0 not found: ID does not exist"
Jul 29 10:58:16.448064 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448037    2551 scope.go:117] "RemoveContainer" containerID="fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"
Jul 29 10:58:16.448278 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.448260    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd\": container with ID starting with fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd not found: ID does not exist" containerID="fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"
Jul 29 10:58:16.448345 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448288    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd"} err="failed to get container status \"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd\": rpc error: code = NotFound desc = could not find container \"fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd\": container with ID starting with fa853196424fc49abb931851b5262cddf15d03c5c97267828157a71add7ac4bd not found: ID does not exist"
Jul 29 10:58:16.448345 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448312    2551 scope.go:117] "RemoveContainer" containerID="f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"
Jul 29 10:58:16.448543 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.448528    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4\": container with ID starting with f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4 not found: ID does not exist" containerID="f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"
Jul 29 10:58:16.448591 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448549    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4"} err="failed to get container status \"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4\": rpc error: code = NotFound desc = could not find container \"f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4\": container with ID starting with f9126d43e3657a0123412709f30840f4bdfa86e4ecfbec907ff3344e967b73b4 not found: ID does not exist"
Jul 29 10:58:16.448591 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.448562    2551 scope.go:117] "RemoveContainer" containerID="f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"
Jul 29 10:58:16.455236 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.455217    2551 scope.go:117] "RemoveContainer" containerID="8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"
Jul 29 10:58:16.461902 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.461887    2551 scope.go:117] "RemoveContainer" containerID="eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"
Jul 29 10:58:16.468454 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.468435    2551 scope.go:117] "RemoveContainer" containerID="f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"
Jul 29 10:58:16.468697 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.468678    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9\": container with ID starting with f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9 not found: ID does not exist" containerID="f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"
Jul 29 10:58:16.468776 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.468706    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9"} err="failed to get container status \"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9\": rpc error: code = NotFound desc = could not find container \"f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9\": container with ID starting with f3470cbec58e30af4b0755017432c75dbf8004e4438b656a3f664e479b28d1e9 not found: ID does not exist"
Jul 29 10:58:16.468776 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.468728    2551 scope.go:117] "RemoveContainer" containerID="8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"
Jul 29 10:58:16.468972 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.468956    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3\": container with ID starting with 8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3 not found: ID does not exist" containerID="8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"
Jul 29 10:58:16.469018 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.468978    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3"} err="failed to get container status \"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3\": rpc error: code = NotFound desc = could not find container \"8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3\": container with ID starting with 8ec252463ac7caaa211a32649c0a7d1849a1b8cf49900b15c094d2d7297ce8e3 not found: ID does not exist"
Jul 29 10:58:16.469018 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.468995    2551 scope.go:117] "RemoveContainer" containerID="eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"
Jul 29 10:58:16.469289 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:16.469269    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1\": container with ID starting with eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1 not found: ID does not exist" containerID="eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"
Jul 29 10:58:16.469356 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.469296    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1"} err="failed to get container status \"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1\": rpc error: code = NotFound desc = could not find container \"eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1\": container with ID starting with eb94331d90ce46b7a7597a3bf34c9e347b62e59c5dbe2ca0dbef069a5ee07df1 not found: ID does not exist"
Jul 29 10:58:16.509638 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509617    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config\") pod \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") "
Jul 29 10:58:16.509720 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509666    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location\") pod \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") "
Jul 29 10:58:16.509720 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509709    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls\") pod \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") "
Jul 29 10:58:16.509809 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509730    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert\") pod \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") "
Jul 29 10:58:16.509809 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509749    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config\") pod \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") "
Jul 29 10:58:16.509809 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509767    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert\") pod \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") "
Jul 29 10:58:16.509954 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509818    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls\") pod \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") "
Jul 29 10:58:16.509954 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509850    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hsjz\" (UniqueName: \"kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz\") pod \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\" (UID: \"cd1321a3-8c2a-4904-aa06-96e6bca7b60f\") "
Jul 29 10:58:16.509954 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509881    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqhxc\" (UniqueName: \"kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc\") pod \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") "
Jul 29 10:58:16.509954 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.509927    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location\") pod \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\" (UID: \"ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7\") "
Jul 29 10:58:16.510182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510058    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" (UID: "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:16.510182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510069    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd1321a3-8c2a-4904-aa06-96e6bca7b60f" (UID: "cd1321a3-8c2a-4904-aa06-96e6bca7b60f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:16.510182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510093    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd1321a3-8c2a-4904-aa06-96e6bca7b60f" (UID: "cd1321a3-8c2a-4904-aa06-96e6bca7b60f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:16.510182 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510154    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd1321a3-8c2a-4904-aa06-96e6bca7b60f" (UID: "cd1321a3-8c2a-4904-aa06-96e6bca7b60f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:16.510385 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510360    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.510425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510383    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.510425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510397    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.510425 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510410    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.510558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510432    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" (UID: "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:16.510558 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.510460    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" (UID: "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:58:16.511765 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.511739    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" (UID: "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:16.511965 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.511942    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc" (OuterVolumeSpecName: "kube-api-access-lqhxc") pod "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" (UID: "ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7"). InnerVolumeSpecName "kube-api-access-lqhxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:16.512018 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.511982    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd1321a3-8c2a-4904-aa06-96e6bca7b60f" (UID: "cd1321a3-8c2a-4904-aa06-96e6bca7b60f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:16.512100 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.512081    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz" (OuterVolumeSpecName: "kube-api-access-4hsjz") pod "cd1321a3-8c2a-4904-aa06-96e6bca7b60f" (UID: "cd1321a3-8c2a-4904-aa06-96e6bca7b60f"). InnerVolumeSpecName "kube-api-access-4hsjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611183    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611203    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hsjz\" (UniqueName: \"kubernetes.io/projected/cd1321a3-8c2a-4904-aa06-96e6bca7b60f-kube-api-access-4hsjz\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611212    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhxc\" (UniqueName: \"kubernetes.io/projected/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kube-api-access-lqhxc\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611219    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611228    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.611234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.611236    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:58:16.748316 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.748287    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:58:16.750433 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.750414    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75f789cb99-5npt5"]
Jul 29 10:58:16.760876 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.760853    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:58:16.768011 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:16.767991    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5d59fd866b-jxtpx"]
Jul 29 10:58:17.296685 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.296647    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:58:17.297004 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.296991    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="storage-initializer"
Jul 29 10:58:17.297056 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297008    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="storage-initializer"
Jul 29 10:58:17.297056 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297022    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container"
Jul 29 10:58:17.297056 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297027    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297037    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="storage-initializer"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297087    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="storage-initializer"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297116    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297122    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297150    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297156    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297163    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container"
Jul 29 10:58:17.297188 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297171    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container"
Jul 29 10:58:17.297453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297274    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kserve-container"
Jul 29 10:58:17.297453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297282    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297288    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" containerName="kube-rbac-proxy"
Jul 29 10:58:17.297453 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.297301    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" containerName="kserve-container"
Jul 29 10:58:17.302402 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.302385    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.305398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.305369    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 29 10:58:17.305541 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.305462    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 10:58:17.305714 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.305695    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 10:58:17.305829 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.305720    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 10:58:17.305829 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.305731    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 10:58:17.306315 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.306301    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 10:58:17.306315 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.306309    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 29 10:58:17.312611 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.312590    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:58:17.321024 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.321003    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 10:58:17.324285 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.324270    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.326820 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.326798    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 29 10:58:17.337284 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.337259    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 10:58:17.417895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.417803    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.417895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.417842    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pp6w\" (UniqueName: \"kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.417895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.417874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.417895 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.417892    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.417980    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.418044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.418096    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsth\" (UniqueName: \"kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.418151    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.418183    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.418398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.418204    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519172 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519172 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519210    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsth\" (UniqueName: \"kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519368    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519405 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519403    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519429    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:17.519446    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:58:17.519533    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls podName:938d20b6-5959-4757-bf76-81ddf6a6b1df nodeName:}" failed. No retries permitted until 2026-07-29 10:58:18.019507203 +0000 UTC m=+1749.921735587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls") pod "isvc-canary-promote-predictor-69b9b94c74-9pq2l" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519558    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519606    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pp6w\" (UniqueName: \"kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.519668 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519624    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.519992 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519977    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.520058 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.519988    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.520116 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.520097    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.520195 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.520173    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.520308 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.520290    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.521989 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.521975    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.530322 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.530295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pp6w\" (UniqueName: \"kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:17.530415 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.530383    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsth\" (UniqueName: \"kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth\") pod \"isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.640395 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.640372    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:17.771224 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:17.771183    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 10:58:17.773294 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:58:17.773264    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e01de3_7b90_45b6_a5cd_3b5712d25f79.slice/crio-be9ed9c424e1611964d6eda5c73ea4221da070fd6e6c28f2c2d983fa8db127ba WatchSource:0}: Error finding container be9ed9c424e1611964d6eda5c73ea4221da070fd6e6c28f2c2d983fa8db127ba: Status 404 returned error can't find the container with id be9ed9c424e1611964d6eda5c73ea4221da070fd6e6c28f2c2d983fa8db127ba
Jul 29 10:58:18.025234 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.025197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:18.027528 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.027508    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") pod \"isvc-canary-promote-predictor-69b9b94c74-9pq2l\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:18.213508 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.213469    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:18.350257 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.350227    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:58:18.351660 ip-10-0-134-91 kubenswrapper[2551]: W0729 10:58:18.351620    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938d20b6_5959_4757_bf76_81ddf6a6b1df.slice/crio-f60ab312e45c83eccf1673b03a495d0eaad1a77731ac873f050faddc612ac94c WatchSource:0}: Error finding container f60ab312e45c83eccf1673b03a495d0eaad1a77731ac873f050faddc612ac94c: Status 404 returned error can't find the container with id f60ab312e45c83eccf1673b03a495d0eaad1a77731ac873f050faddc612ac94c
Jul 29 10:58:18.437757 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.437725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerStarted","Data":"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"}
Jul 29 10:58:18.438147 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.437790    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerStarted","Data":"be9ed9c424e1611964d6eda5c73ea4221da070fd6e6c28f2c2d983fa8db127ba"}
Jul 29 10:58:18.439547 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.439523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerStarted","Data":"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"}
Jul 29 10:58:18.439648 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.439581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerStarted","Data":"f60ab312e45c83eccf1673b03a495d0eaad1a77731ac873f050faddc612ac94c"}
Jul 29 10:58:18.729643 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.729615    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1321a3-8c2a-4904-aa06-96e6bca7b60f" path="/var/lib/kubelet/pods/cd1321a3-8c2a-4904-aa06-96e6bca7b60f/volumes"
Jul 29 10:58:18.730081 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:18.730068    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7" path="/var/lib/kubelet/pods/ef9e6bf2-56b1-4a81-a076-eaa8cdef97d7/volumes"
Jul 29 10:58:22.462217 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:22.462185    2551 generic.go:356] "Generic (PLEG): container finished" podID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerID="02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a" exitCode=0
Jul 29 10:58:22.462619 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:22.462258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerDied","Data":"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"}
Jul 29 10:58:22.463956 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:22.463936    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerID="85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930" exitCode=0
Jul 29 10:58:22.464044 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:22.463968    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerDied","Data":"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"}
Jul 29 10:58:23.470149 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.470100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerStarted","Data":"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"}
Jul 29 10:58:23.470617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.470183    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerStarted","Data":"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"}
Jul 29 10:58:23.470617 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.470439    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:23.472279 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.472255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerStarted","Data":"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"}
Jul 29 10:58:23.472398 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.472301    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerStarted","Data":"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"}
Jul 29 10:58:23.472487 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.472472    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:23.490972 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.490932    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podStartSLOduration=2.745838316 podStartE2EDuration="6.490918194s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745079878s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:23.489597813 +0000 UTC m=+1755.391826223" watchObservedRunningTime="2026-07-29 10:58:23.490918194 +0000 UTC m=+1755.393146630"
Jul 29 10:58:23.508486 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:23.508440    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podStartSLOduration=2.929516469 podStartE2EDuration="6.508426793s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.578910324s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:23.507614472 +0000 UTC m=+1755.409842870" watchObservedRunningTime="2026-07-29 10:58:23.508426793 +0000 UTC m=+1755.410655189"
Jul 29 10:58:24.477510 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:24.477474    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:24.477919 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:24.477537    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:24.478549 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:24.478522    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:58:24.478657 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:24.478552    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:58:25.482051 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:25.482008    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:58:25.482555 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:25.482531    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:58:30.487078 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:30.487047    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:58:30.487646 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:30.487617    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:58:30.488840 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:30.488820    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:58:30.489387 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:30.489354    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:58:40.487909 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:40.487867    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:58:40.490163 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:40.490123    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:58:50.488348 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:50.488309    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:58:50.489581 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:58:50.489556    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:59:00.488477 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:00.488383    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:59:00.489637 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:00.489607    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:59:08.688009 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:08.687977    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:59:08.691773 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:08.691752    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 10:59:10.487618 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:10.487581    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:59:10.489926 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:10.489886    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:59:20.488457 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:20.488416    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:59:20.489641 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:20.489611    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:59:30.488097 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:30.488056    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:59:30.489280 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:30.489258    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 10:59:38.727968 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:38.727935    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 10:59:40.488096 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:40.488068    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:59:47.299302 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:47.299270    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:59:47.299718 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:47.299494    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" containerID="cri-o://8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7" gracePeriod=30
Jul 29 10:59:47.299718 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:47.299579    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kube-rbac-proxy" containerID="cri-o://d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56" gracePeriod=30
Jul 29 10:59:47.847753 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:47.847723    2551 generic.go:356] "Generic (PLEG): container finished" podID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerID="d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56" exitCode=2
Jul 29 10:59:47.847919 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:47.847800    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerDied","Data":"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"}
Jul 29 10:59:50.482522 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:50.482481    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 29 10:59:50.488440 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:50.488404    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 29 10:59:51.042505 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.042479    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:59:51.117891 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.117863    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert\") pod \"938d20b6-5959-4757-bf76-81ddf6a6b1df\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") "
Jul 29 10:59:51.118049 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.117901    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location\") pod \"938d20b6-5959-4757-bf76-81ddf6a6b1df\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") "
Jul 29 10:59:51.118049 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.117923    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pp6w\" (UniqueName: \"kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w\") pod \"938d20b6-5959-4757-bf76-81ddf6a6b1df\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") "
Jul 29 10:59:51.118224 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118046    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config\") pod \"938d20b6-5959-4757-bf76-81ddf6a6b1df\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") "
Jul 29 10:59:51.118224 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118096    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") pod \"938d20b6-5959-4757-bf76-81ddf6a6b1df\" (UID: \"938d20b6-5959-4757-bf76-81ddf6a6b1df\") "
Jul 29 10:59:51.118224 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118202    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "938d20b6-5959-4757-bf76-81ddf6a6b1df" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:51.118376 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118267    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "938d20b6-5959-4757-bf76-81ddf6a6b1df" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:51.118421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118378    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "938d20b6-5959-4757-bf76-81ddf6a6b1df" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:59:51.118464 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118440    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/938d20b6-5959-4757-bf76-81ddf6a6b1df-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:59:51.118464 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.118456    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:59:51.119997 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.119969    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "938d20b6-5959-4757-bf76-81ddf6a6b1df" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:51.120092 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.120055    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w" (OuterVolumeSpecName: "kube-api-access-4pp6w") pod "938d20b6-5959-4757-bf76-81ddf6a6b1df" (UID: "938d20b6-5959-4757-bf76-81ddf6a6b1df"). InnerVolumeSpecName "kube-api-access-4pp6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:51.219105 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.219071    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pp6w\" (UniqueName: \"kubernetes.io/projected/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-api-access-4pp6w\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:59:51.219105 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.219093    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/938d20b6-5959-4757-bf76-81ddf6a6b1df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:59:51.219105 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.219103    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/938d20b6-5959-4757-bf76-81ddf6a6b1df-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 10:59:51.866482 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.866450    2551 generic.go:356] "Generic (PLEG): container finished" podID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerID="8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7" exitCode=0
Jul 29 10:59:51.866882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.866502    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"
Jul 29 10:59:51.866882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.866537    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerDied","Data":"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"}
Jul 29 10:59:51.866882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.866595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l" event={"ID":"938d20b6-5959-4757-bf76-81ddf6a6b1df","Type":"ContainerDied","Data":"f60ab312e45c83eccf1673b03a495d0eaad1a77731ac873f050faddc612ac94c"}
Jul 29 10:59:51.866882 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.866625    2551 scope.go:117] "RemoveContainer" containerID="d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"
Jul 29 10:59:51.876014 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.875997    2551 scope.go:117] "RemoveContainer" containerID="8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"
Jul 29 10:59:51.883163 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.883148    2551 scope.go:117] "RemoveContainer" containerID="02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"
Jul 29 10:59:51.889421 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.889395    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:59:51.890777 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.890762    2551 scope.go:117] "RemoveContainer" containerID="d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"
Jul 29 10:59:51.891039 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:59:51.891023    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56\": container with ID starting with d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56 not found: ID does not exist" containerID="d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"
Jul 29 10:59:51.891084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.891047    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56"} err="failed to get container status \"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56\": rpc error: code = NotFound desc = could not find container \"d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56\": container with ID starting with d11e21b54b319bb8eb5536db40386d8b94d9a6cf4cb4450b4888e5d5c4ad1a56 not found: ID does not exist"
Jul 29 10:59:51.891084 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.891066    2551 scope.go:117] "RemoveContainer" containerID="8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"
Jul 29 10:59:51.891323 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:59:51.891309    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7\": container with ID starting with 8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7 not found: ID does not exist" containerID="8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"
Jul 29 10:59:51.891385 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.891326    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7"} err="failed to get container status \"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7\": rpc error: code = NotFound desc = could not find container \"8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7\": container with ID starting with 8a91aaab53cc370a6e138da74750fe79b5c034ee329be834c069824d576088b7 not found: ID does not exist"
Jul 29 10:59:51.891385 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.891339    2551 scope.go:117] "RemoveContainer" containerID="02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"
Jul 29 10:59:51.891598 ip-10-0-134-91 kubenswrapper[2551]: E0729 10:59:51.891577    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a\": container with ID starting with 02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a not found: ID does not exist" containerID="02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"
Jul 29 10:59:51.891657 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.891600    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a"} err="failed to get container status \"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a\": rpc error: code = NotFound desc = could not find container \"02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a\": container with ID starting with 02dd3f5cdbf96ef194fbe67c81a940608857b538a991f51f8ac5eee22f5b1f8a not found: ID does not exist"
Jul 29 10:59:51.894890 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:51.894858    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-69b9b94c74-9pq2l"]
Jul 29 10:59:52.728288 ip-10-0-134-91 kubenswrapper[2551]: I0729 10:59:52.728255    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" path="/var/lib/kubelet/pods/938d20b6-5959-4757-bf76-81ddf6a6b1df/volumes"
Jul 29 11:01:27.551309 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:27.551274    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 11:01:27.554027 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:27.551560    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" containerID="cri-o://bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04" gracePeriod=30
Jul 29 11:01:27.554027 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:27.551581    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kube-rbac-proxy" containerID="cri-o://f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598" gracePeriod=30
Jul 29 11:01:28.286669 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:28.286636    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerID="f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598" exitCode=2
Jul 29 11:01:28.286883 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:28.286717    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerDied","Data":"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"}
Jul 29 11:01:28.725088 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:28.725049    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 29 11:01:30.482873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:30.482836    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 29 11:01:31.591018 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.590988    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 11:01:31.660716 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.660659    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls\") pod \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") "
Jul 29 11:01:31.660716 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.660709    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config\") pod \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") "
Jul 29 11:01:31.660915 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.660756    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcsth\" (UniqueName: \"kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth\") pod \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") "
Jul 29 11:01:31.660915 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.660783    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert\") pod \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") "
Jul 29 11:01:31.660915 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.660825    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location\") pod \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\" (UID: \"d4e01de3-7b90-45b6-a5cd-3b5712d25f79\") "
Jul 29 11:01:31.661148 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.661109    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4e01de3-7b90-45b6-a5cd-3b5712d25f79" (UID: "d4e01de3-7b90-45b6-a5cd-3b5712d25f79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:01:31.661206 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.661175    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4e01de3-7b90-45b6-a5cd-3b5712d25f79" (UID: "d4e01de3-7b90-45b6-a5cd-3b5712d25f79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:01:31.661206 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.661180    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4e01de3-7b90-45b6-a5cd-3b5712d25f79" (UID: "d4e01de3-7b90-45b6-a5cd-3b5712d25f79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:01:31.662568 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.662551    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4e01de3-7b90-45b6-a5cd-3b5712d25f79" (UID: "d4e01de3-7b90-45b6-a5cd-3b5712d25f79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:01:31.662657 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.662626    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth" (OuterVolumeSpecName: "kube-api-access-dcsth") pod "d4e01de3-7b90-45b6-a5cd-3b5712d25f79" (UID: "d4e01de3-7b90-45b6-a5cd-3b5712d25f79"). InnerVolumeSpecName "kube-api-access-dcsth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:01:31.762202 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.762181    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:01:31.762202 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.762201    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:01:31.762320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.762212    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:01:31.762320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.762220    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsth\" (UniqueName: \"kubernetes.io/projected/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-kube-api-access-dcsth\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:01:31.762320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:31.762229    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4e01de3-7b90-45b6-a5cd-3b5712d25f79-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:01:32.306072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.306040    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerID="bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04" exitCode=0
Jul 29 11:01:32.306236 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.306113    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"
Jul 29 11:01:32.306236 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.306156    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerDied","Data":"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"}
Jul 29 11:01:32.306236 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.306232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz" event={"ID":"d4e01de3-7b90-45b6-a5cd-3b5712d25f79","Type":"ContainerDied","Data":"be9ed9c424e1611964d6eda5c73ea4221da070fd6e6c28f2c2d983fa8db127ba"}
Jul 29 11:01:32.306378 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.306262    2551 scope.go:117] "RemoveContainer" containerID="f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"
Jul 29 11:01:32.316111 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.316091    2551 scope.go:117] "RemoveContainer" containerID="bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"
Jul 29 11:01:32.324562 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.324538    2551 scope.go:117] "RemoveContainer" containerID="85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"
Jul 29 11:01:32.329284 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.329259    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 11:01:32.332459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.332436    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-869c8cbf4f-n48cz"]
Jul 29 11:01:32.333265 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.333250    2551 scope.go:117] "RemoveContainer" containerID="f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"
Jul 29 11:01:32.333552 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.333533    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598\": container with ID starting with f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598 not found: ID does not exist" containerID="f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"
Jul 29 11:01:32.333608 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.333559    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598"} err="failed to get container status \"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598\": rpc error: code = NotFound desc = could not find container \"f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598\": container with ID starting with f698f833dde60dbb52787ae50e8f29e8111eb538bb817964149c5ca242eff598 not found: ID does not exist"
Jul 29 11:01:32.333608 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.333578    2551 scope.go:117] "RemoveContainer" containerID="bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"
Jul 29 11:01:32.333811 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.333792    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04\": container with ID starting with bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04 not found: ID does not exist" containerID="bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"
Jul 29 11:01:32.333852 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.333816    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04"} err="failed to get container status \"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04\": rpc error: code = NotFound desc = could not find container \"bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04\": container with ID starting with bad3373ad90cab40d02944c746cf69417e818e1b0901225844a8f885ccfa4c04 not found: ID does not exist"
Jul 29 11:01:32.333852 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.333831    2551 scope.go:117] "RemoveContainer" containerID="85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"
Jul 29 11:01:32.334094 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.334075    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930\": container with ID starting with 85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930 not found: ID does not exist" containerID="85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"
Jul 29 11:01:32.334188 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.334096    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930"} err="failed to get container status \"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930\": rpc error: code = NotFound desc = could not find container \"85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930\": container with ID starting with 85cf0bf60f31234631b5de8bf6b4d71e74e4790f80e4f3aab9d5dbd03832c930 not found: ID does not exist"
Jul 29 11:01:32.728089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728063    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" path="/var/lib/kubelet/pods/d4e01de3-7b90-45b6-a5cd-3b5712d25f79/volumes"
Jul 29 11:01:32.728440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728402    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:01:32.728703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728690    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container"
Jul 29 11:01:32.728744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728705    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container"
Jul 29 11:01:32.728744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728716    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="storage-initializer"
Jul 29 11:01:32.728744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728723    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="storage-initializer"
Jul 29 11:01:32.728744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728731    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="storage-initializer"
Jul 29 11:01:32.728744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728736    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="storage-initializer"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728749    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728755    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728765    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728770    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728780    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728784    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728829    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kserve-container"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728836    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="938d20b6-5959-4757-bf76-81ddf6a6b1df" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728845    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kube-rbac-proxy"
Jul 29 11:01:32.728889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.728850    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e01de3-7b90-45b6-a5cd-3b5712d25f79" containerName="kserve-container"
Jul 29 11:01:32.733682 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.733664    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.739942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.739764    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 29 11:01:32.740228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.740207    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:01:32.740349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.740207    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:01:32.740349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.740270    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:01:32.740623 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.740608    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:01:32.741020 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.741006    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:01:32.747030 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.747015    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 29 11:01:32.765329 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.765307    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:01:32.766677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.766659    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:01:32.767656 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.767633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6jw\" (UniqueName: \"kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.767756 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.767693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.767756 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.767729    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.767870 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.767757    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.767870 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.767806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.770311 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.770295    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.776531 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.776515    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 29 11:01:32.793021 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.793004    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:01:32.868186 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868193    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868225    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868247    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.868303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868272    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6jw\" (UniqueName: \"kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.868336    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868404    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xt7h\" (UniqueName: \"kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.868421    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls podName:bad8e73f-a406-4f46-a240-4eefdab2af16 nodeName:}" failed. No retries permitted until 2026-07-29 11:01:33.368404473 +0000 UTC m=+1945.270632861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls") pod "isvc-canary-rollback-predictor-79b59b5966-rbckt" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868447    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868479    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868514    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.868906 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868539    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868968 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868917    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.868968 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.868953    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.876566 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.876547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6jw\" (UniqueName: \"kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:32.969383 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969452    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xt7h\" (UniqueName: \"kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969481    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.969660    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.969749 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.969732    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 29 11:01:32.969805 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:01:32.969800    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls podName:6f067f8f-230e-4943-a4e4-6e050a344aaa nodeName:}" failed. No retries permitted until 2026-07-29 11:01:33.469780553 +0000 UTC m=+1945.372008943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 29 11:01:32.970084 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.970065    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.970168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.970151    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:32.977810 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:32.977784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xt7h\" (UniqueName: \"kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:33.372014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.371986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:33.374264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.374245    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") pod \"isvc-canary-rollback-predictor-79b59b5966-rbckt\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:33.472454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.472425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:33.474570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.474551    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:33.643592 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.643517    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:33.678933 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.678904    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:33.794107 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.794082    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:01:33.796077 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:01:33.796051    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad8e73f_a406_4f46_a240_4eefdab2af16.slice/crio-90315844510ed8683bf421950ce04aa64b81e07cef905df26083d45822440e5f WatchSource:0}: Error finding container 90315844510ed8683bf421950ce04aa64b81e07cef905df26083d45822440e5f: Status 404 returned error can't find the container with id 90315844510ed8683bf421950ce04aa64b81e07cef905df26083d45822440e5f
Jul 29 11:01:33.812187 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:33.812167    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:01:33.814351 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:01:33.814330    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f067f8f_230e_4943_a4e4_6e050a344aaa.slice/crio-7440bf61c4308ad41ab0664b525c7748da029650fc2d717aa56327cccc51f81a WatchSource:0}: Error finding container 7440bf61c4308ad41ab0664b525c7748da029650fc2d717aa56327cccc51f81a: Status 404 returned error can't find the container with id 7440bf61c4308ad41ab0664b525c7748da029650fc2d717aa56327cccc51f81a
Jul 29 11:01:34.318480 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:34.318432    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerStarted","Data":"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"}
Jul 29 11:01:34.318682 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:34.318541    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerStarted","Data":"7440bf61c4308ad41ab0664b525c7748da029650fc2d717aa56327cccc51f81a"}
Jul 29 11:01:34.327448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:34.327408    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerStarted","Data":"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"}
Jul 29 11:01:34.327594 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:34.327513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerStarted","Data":"90315844510ed8683bf421950ce04aa64b81e07cef905df26083d45822440e5f"}
Jul 29 11:01:38.351659 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:38.351624    2551 generic.go:356] "Generic (PLEG): container finished" podID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerID="2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc" exitCode=0
Jul 29 11:01:38.352158 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:38.351708    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerDied","Data":"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"}
Jul 29 11:01:38.353730 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:38.353707    2551 generic.go:356] "Generic (PLEG): container finished" podID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerID="563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582" exitCode=0
Jul 29 11:01:38.353826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:38.353794    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerDied","Data":"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"}
Jul 29 11:01:39.360383 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.360345    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerStarted","Data":"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"}
Jul 29 11:01:39.360838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.360415    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerStarted","Data":"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"}
Jul 29 11:01:39.360838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.360768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:39.360997 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.360944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:39.362310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.362280    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:01:39.362750 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.362731    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerStarted","Data":"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"}
Jul 29 11:01:39.362817 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.362776    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerStarted","Data":"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"}
Jul 29 11:01:39.362953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.362938    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:39.406950 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.406906    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podStartSLOduration=3.816920315 podStartE2EDuration="7.406892607s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.589972292s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:01:39.405505236 +0000 UTC m=+1951.307733634" watchObservedRunningTime="2026-07-29 11:01:39.406892607 +0000 UTC m=+1951.309121004"
Jul 29 11:01:39.407068 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:39.407001    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podStartSLOduration=3.538936757 podStartE2EDuration="7.406996498s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868059741s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:01:39.385657947 +0000 UTC m=+1951.287886358" watchObservedRunningTime="2026-07-29 11:01:39.406996498 +0000 UTC m=+1951.309224892"
Jul 29 11:01:40.367706 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:40.367665    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:01:40.368185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:40.367901    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:40.368902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:40.368870    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:01:41.372703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:41.372663    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:01:41.373175 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:41.372803    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:01:46.377181 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:46.377149    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:01:46.377970 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:46.377921    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:01:46.378241 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:46.378219    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:01:46.378748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:46.378722    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:01:56.377831 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:56.377789    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:01:56.379058 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:01:56.379027    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:06.377807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:06.377718    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:02:06.378951 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:06.378921    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:16.377814 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:16.377771    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:02:16.379044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:16.379017    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:26.378280 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:26.378239    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:02:26.378735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:26.378647    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:36.378013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:36.377973    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:02:36.379239 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:36.379207    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:46.378555 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:46.378517    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 29 11:02:46.379010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:46.378649    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:02:48.727885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:48.727858    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:02:56.378320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:56.378283    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:02:57.719646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:57.719612    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:02:57.720032 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:57.719848    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container" containerID="cri-o://6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0" gracePeriod=30
Jul 29 11:02:57.720032 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:57.719881    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kube-rbac-proxy" containerID="cri-o://6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879" gracePeriod=30
Jul 29 11:02:58.708579 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:58.708542    2551 generic.go:356] "Generic (PLEG): container finished" podID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerID="6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879" exitCode=2
Jul 29 11:02:58.708745 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:02:58.708586    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerDied","Data":"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"}
Jul 29 11:03:01.373833 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.373789    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 29 11:03:01.657563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.657540    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:03:01.719889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.719863    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") pod \"6f067f8f-230e-4943-a4e4-6e050a344aaa\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") "
Jul 29 11:03:01.720015 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.719949    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config\") pod \"6f067f8f-230e-4943-a4e4-6e050a344aaa\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") "
Jul 29 11:03:01.720015 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.719987    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location\") pod \"6f067f8f-230e-4943-a4e4-6e050a344aaa\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") "
Jul 29 11:03:01.720156 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.720015    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert\") pod \"6f067f8f-230e-4943-a4e4-6e050a344aaa\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") "
Jul 29 11:03:01.720156 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.720043    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xt7h\" (UniqueName: \"kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h\") pod \"6f067f8f-230e-4943-a4e4-6e050a344aaa\" (UID: \"6f067f8f-230e-4943-a4e4-6e050a344aaa\") "
Jul 29 11:03:01.720367 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.720338    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f067f8f-230e-4943-a4e4-6e050a344aaa" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:03:01.720367 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.720354    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f067f8f-230e-4943-a4e4-6e050a344aaa" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:03:01.720490 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.720350    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f067f8f-230e-4943-a4e4-6e050a344aaa" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:03:01.721570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.721539    2551 generic.go:356] "Generic (PLEG): container finished" podID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerID="6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0" exitCode=0
Jul 29 11:03:01.721679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.721603    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"
Jul 29 11:03:01.721679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.721616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerDied","Data":"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"}
Jul 29 11:03:01.721798 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.721695    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc" event={"ID":"6f067f8f-230e-4943-a4e4-6e050a344aaa","Type":"ContainerDied","Data":"7440bf61c4308ad41ab0664b525c7748da029650fc2d717aa56327cccc51f81a"}
Jul 29 11:03:01.721798 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.721735    2551 scope.go:117] "RemoveContainer" containerID="6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"
Jul 29 11:03:01.722123 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.722100    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f067f8f-230e-4943-a4e4-6e050a344aaa" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:03:01.722273 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.722247    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h" (OuterVolumeSpecName: "kube-api-access-6xt7h") pod "6f067f8f-230e-4943-a4e4-6e050a344aaa" (UID: "6f067f8f-230e-4943-a4e4-6e050a344aaa"). InnerVolumeSpecName "kube-api-access-6xt7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:03:01.736145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.736116    2551 scope.go:117] "RemoveContainer" containerID="6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"
Jul 29 11:03:01.745027 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.745009    2551 scope.go:117] "RemoveContainer" containerID="2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"
Jul 29 11:03:01.751934 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.751918    2551 scope.go:117] "RemoveContainer" containerID="6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"
Jul 29 11:03:01.752201 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:03:01.752185    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879\": container with ID starting with 6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879 not found: ID does not exist" containerID="6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"
Jul 29 11:03:01.752254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.752208    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879"} err="failed to get container status \"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879\": rpc error: code = NotFound desc = could not find container \"6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879\": container with ID starting with 6b8340dc579970070a2dc79b674a21612c99fe0a506cdb6f8e43e2de62bb1879 not found: ID does not exist"
Jul 29 11:03:01.752254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.752225    2551 scope.go:117] "RemoveContainer" containerID="6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"
Jul 29 11:03:01.752470 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:03:01.752452    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0\": container with ID starting with 6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0 not found: ID does not exist" containerID="6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"
Jul 29 11:03:01.752518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.752477    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0"} err="failed to get container status \"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0\": rpc error: code = NotFound desc = could not find container \"6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0\": container with ID starting with 6f316d41502f0c624f1357ca3784646d89e95f898058dae2f94e668b6bab30c0 not found: ID does not exist"
Jul 29 11:03:01.752518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.752493    2551 scope.go:117] "RemoveContainer" containerID="2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"
Jul 29 11:03:01.752714 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:03:01.752698    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc\": container with ID starting with 2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc not found: ID does not exist" containerID="2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"
Jul 29 11:03:01.752757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.752716    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc"} err="failed to get container status \"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc\": rpc error: code = NotFound desc = could not find container \"2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc\": container with ID starting with 2c3bb185fe34f5468a9172fecc2ba6bf0073ca732db6af3dbb1dab6d3055aefc not found: ID does not exist"
Jul 29 11:03:01.820759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.820737    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:03:01.820759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.820757    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f067f8f-230e-4943-a4e4-6e050a344aaa-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:03:01.820873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.820766    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f067f8f-230e-4943-a4e4-6e050a344aaa-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:03:01.820873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.820776    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xt7h\" (UniqueName: \"kubernetes.io/projected/6f067f8f-230e-4943-a4e4-6e050a344aaa-kube-api-access-6xt7h\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:03:01.820873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:01.820785    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f067f8f-230e-4943-a4e4-6e050a344aaa-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:03:02.045177 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:02.045144    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:03:02.048602 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:02.048580    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b74c55879-w7vzc"]
Jul 29 11:03:02.728487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:03:02.728451    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" path="/var/lib/kubelet/pods/6f067f8f-230e-4943-a4e4-6e050a344aaa/volumes"
Jul 29 11:04:08.715006 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:08.714978    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:04:08.720208 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:08.720186    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:04:27.918847 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:27.918811    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:04:27.919298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:27.919092    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" containerID="cri-o://1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f" gracePeriod=30
Jul 29 11:04:27.919298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:27.919114    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kube-rbac-proxy" containerID="cri-o://e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9" gracePeriod=30
Jul 29 11:04:28.087375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:28.087349    2551 generic.go:356] "Generic (PLEG): container finished" podID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerID="e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9" exitCode=2
Jul 29 11:04:28.087485 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:28.087428    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerDied","Data":"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"}
Jul 29 11:04:28.725718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:28.725679    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 29 11:04:31.373773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.373726    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 29 11:04:31.861562 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.861537    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:04:31.983312 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983290    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt6jw\" (UniqueName: \"kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw\") pod \"bad8e73f-a406-4f46-a240-4eefdab2af16\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") "
Jul 29 11:04:31.983422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983324    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config\") pod \"bad8e73f-a406-4f46-a240-4eefdab2af16\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") "
Jul 29 11:04:31.983422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983354    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert\") pod \"bad8e73f-a406-4f46-a240-4eefdab2af16\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") "
Jul 29 11:04:31.983499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983425    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") pod \"bad8e73f-a406-4f46-a240-4eefdab2af16\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") "
Jul 29 11:04:31.983499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983462    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location\") pod \"bad8e73f-a406-4f46-a240-4eefdab2af16\" (UID: \"bad8e73f-a406-4f46-a240-4eefdab2af16\") "
Jul 29 11:04:31.983774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983708    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bad8e73f-a406-4f46-a240-4eefdab2af16" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:04:31.983870 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983850    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bad8e73f-a406-4f46-a240-4eefdab2af16" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:31.983953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.983929    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bad8e73f-a406-4f46-a240-4eefdab2af16" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:04:31.985385 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.985364    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw" (OuterVolumeSpecName: "kube-api-access-bt6jw") pod "bad8e73f-a406-4f46-a240-4eefdab2af16" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16"). InnerVolumeSpecName "kube-api-access-bt6jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:04:31.985440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:31.985395    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bad8e73f-a406-4f46-a240-4eefdab2af16" (UID: "bad8e73f-a406-4f46-a240-4eefdab2af16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:04:32.084699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.084676    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6jw\" (UniqueName: \"kubernetes.io/projected/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-api-access-bt6jw\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:04:32.084699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.084700    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:04:32.084818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.084710    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad8e73f-a406-4f46-a240-4eefdab2af16-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:04:32.084818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.084719    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad8e73f-a406-4f46-a240-4eefdab2af16-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:04:32.084818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.084727    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad8e73f-a406-4f46-a240-4eefdab2af16-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:04:32.104759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.104734    2551 generic.go:356] "Generic (PLEG): container finished" podID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerID="1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f" exitCode=0
Jul 29 11:04:32.104851 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.104783    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"
Jul 29 11:04:32.104851 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.104819    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerDied","Data":"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"}
Jul 29 11:04:32.104925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.104902    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt" event={"ID":"bad8e73f-a406-4f46-a240-4eefdab2af16","Type":"ContainerDied","Data":"90315844510ed8683bf421950ce04aa64b81e07cef905df26083d45822440e5f"}
Jul 29 11:04:32.104967 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.104932    2551 scope.go:117] "RemoveContainer" containerID="e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"
Jul 29 11:04:32.113722 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.113704    2551 scope.go:117] "RemoveContainer" containerID="1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"
Jul 29 11:04:32.120718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.120702    2551 scope.go:117] "RemoveContainer" containerID="563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"
Jul 29 11:04:32.127593 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.127569    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:04:32.128910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.128893    2551 scope.go:117] "RemoveContainer" containerID="e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"
Jul 29 11:04:32.129358 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:04:32.129331    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9\": container with ID starting with e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9 not found: ID does not exist" containerID="e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"
Jul 29 11:04:32.129465 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.129365    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9"} err="failed to get container status \"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9\": rpc error: code = NotFound desc = could not find container \"e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9\": container with ID starting with e67cb5f0f0746c3a5d5ffa9c3fd9a32c395b8ae932c6112e4eeb1fad6f09bdc9 not found: ID does not exist"
Jul 29 11:04:32.129465 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.129384    2551 scope.go:117] "RemoveContainer" containerID="1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"
Jul 29 11:04:32.129658 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:04:32.129640    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f\": container with ID starting with 1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f not found: ID does not exist" containerID="1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"
Jul 29 11:04:32.129701 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.129663    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f"} err="failed to get container status \"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f\": rpc error: code = NotFound desc = could not find container \"1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f\": container with ID starting with 1cfd0919b9c7ac512100a5a14aae01f722487357d5c62996486e4961a3603d0f not found: ID does not exist"
Jul 29 11:04:32.129701 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.129677    2551 scope.go:117] "RemoveContainer" containerID="563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"
Jul 29 11:04:32.129933 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:04:32.129917    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582\": container with ID starting with 563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582 not found: ID does not exist" containerID="563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"
Jul 29 11:04:32.129983 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.129947    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582"} err="failed to get container status \"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582\": rpc error: code = NotFound desc = could not find container \"563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582\": container with ID starting with 563d182e6cfcf403e1fb2ef00b18e6326aad6cf0b9bbfc922fec91c6cc767582 not found: ID does not exist"
Jul 29 11:04:32.130711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.130692    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-79b59b5966-rbckt"]
Jul 29 11:04:32.728187 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:32.728161    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" path="/var/lib/kubelet/pods/bad8e73f-a406-4f46-a240-4eefdab2af16/volumes"
Jul 29 11:04:33.089432 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089367    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:04:33.089674 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089661    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="storage-initializer"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089675    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="storage-initializer"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089684    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="storage-initializer"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089689    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="storage-initializer"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089698    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089705    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089713    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089718    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089734    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089740    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089749    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089754    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089822    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kserve-container"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089830    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089836    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad8e73f-a406-4f46-a240-4eefdab2af16" containerName="kube-rbac-proxy"
Jul 29 11:04:33.089929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.089843    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f067f8f-230e-4943-a4e4-6e050a344aaa" containerName="kserve-container"
Jul 29 11:04:33.094324 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.094306    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.094497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.094466    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:04:33.096778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.096756    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 29 11:04:33.096887 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.096797    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:04:33.096887 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.096820    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:04:33.096989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.096882    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:04:33.096989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.096956    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 29 11:04:33.097744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.097729    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.097967 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.097947    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:04:33.098055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.098022    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:04:33.099763 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.099743    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 29 11:04:33.103607 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.103586    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:04:33.107340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.107323    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:04:33.193792 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193767    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.193875 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193795    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8dv\" (UniqueName: \"kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.193875 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193815    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.193875 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwllr\" (UniqueName: \"kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.194005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193908    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.194005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193952    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.194005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.193975    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.194110 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.194007    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.194110 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.194045    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.194110 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.194073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294562 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294575    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294596    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294638    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.294658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8dv\" (UniqueName: \"kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294672    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294696    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwllr\" (UniqueName: \"kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:04:33.294759    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294775    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294803    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.294854 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:04:33.294836    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls podName:26ce6cdf-0af2-4665-9fb2-337986959768 nodeName:}" failed. No retries permitted until 2026-07-29 11:04:33.794815826 +0000 UTC m=+2125.697044201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls") pod "isvc-canary-stop-predictor-75dfd479cd-26gh9" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 29 11:04:33.295270 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.294911    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.295335 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.295282    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.295506 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.295485    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.295656 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.295636    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.295696 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.295636    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.295742 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.295720    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.297310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.297292    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.302726 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.302699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8dv\" (UniqueName: \"kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.303042 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.303020    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwllr\" (UniqueName: \"kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr\") pod \"isvc-canary-stop-v2-predictor-865fc6b775-zb88z\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.414747 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.414692    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:33.547895 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.547873    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:04:33.550549 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:04:33.550517    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607ad23e_2acf_4a9d_b271_b21350c7460a.slice/crio-aa2debe57ef8f091cdc4a3c46fe537cf406f8e3b0473a55f2574670e4ebd9b93 WatchSource:0}: Error finding container aa2debe57ef8f091cdc4a3c46fe537cf406f8e3b0473a55f2574670e4ebd9b93: Status 404 returned error can't find the container with id aa2debe57ef8f091cdc4a3c46fe537cf406f8e3b0473a55f2574670e4ebd9b93
Jul 29 11:04:33.799121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.799090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:33.801417 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:33.801397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") pod \"isvc-canary-stop-predictor-75dfd479cd-26gh9\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:34.006777 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:34.006748    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:34.118286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:34.118245    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerStarted","Data":"5a458151681e997271d3599960b347cf47facfd1ff15c8f584fa2524740a7e4e"}
Jul 29 11:04:34.118479 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:34.118320    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerStarted","Data":"aa2debe57ef8f091cdc4a3c46fe537cf406f8e3b0473a55f2574670e4ebd9b93"}
Jul 29 11:04:34.142937 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:34.142906    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:04:34.143827 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:04:34.143797    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ce6cdf_0af2_4665_9fb2_337986959768.slice/crio-bea99c784586180f96999a132eec96189742da48ddabb3f6110677aff747da55 WatchSource:0}: Error finding container bea99c784586180f96999a132eec96189742da48ddabb3f6110677aff747da55: Status 404 returned error can't find the container with id bea99c784586180f96999a132eec96189742da48ddabb3f6110677aff747da55
Jul 29 11:04:35.123702 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:35.123664    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerStarted","Data":"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"}
Jul 29 11:04:35.124115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:35.123738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerStarted","Data":"bea99c784586180f96999a132eec96189742da48ddabb3f6110677aff747da55"}
Jul 29 11:04:38.138243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:38.138213    2551 generic.go:356] "Generic (PLEG): container finished" podID="26ce6cdf-0af2-4665-9fb2-337986959768" containerID="e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0" exitCode=0
Jul 29 11:04:38.138652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:38.138292    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerDied","Data":"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"}
Jul 29 11:04:38.139861 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:38.139835    2551 generic.go:356] "Generic (PLEG): container finished" podID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerID="5a458151681e997271d3599960b347cf47facfd1ff15c8f584fa2524740a7e4e" exitCode=0
Jul 29 11:04:38.139963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:38.139882    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerDied","Data":"5a458151681e997271d3599960b347cf47facfd1ff15c8f584fa2524740a7e4e"}
Jul 29 11:04:39.146229 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.146187    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerStarted","Data":"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"}
Jul 29 11:04:39.146683 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.146255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerStarted","Data":"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"}
Jul 29 11:04:39.146683 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.146456    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:39.148308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.148277    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerStarted","Data":"37d52f7650af6bd47652b058b9211af5b035fca0cd0f6a679d3d260f799ea5e8"}
Jul 29 11:04:39.148433 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.148340    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerStarted","Data":"23b062062cc02640c51f52e98cceeee7dde55376e1f12c4f4cc0265bcd72e9de"}
Jul 29 11:04:39.148628 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.148609    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:39.148714 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.148660    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:39.149999 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.149972    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:04:39.165846 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.165808    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podStartSLOduration=2.353815559 podStartE2EDuration="6.165794625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811979066s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:04:39.165375922 +0000 UTC m=+2131.067604320" watchObservedRunningTime="2026-07-29 11:04:39.165794625 +0000 UTC m=+2131.068023022"
Jul 29 11:04:39.184309 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:39.184258    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podStartSLOduration=2.188431898 podStartE2EDuration="6.184242041s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.995810143s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:04:39.183829391 +0000 UTC m=+2131.086057789" watchObservedRunningTime="2026-07-29 11:04:39.184242041 +0000 UTC m=+2131.086470446"
Jul 29 11:04:40.152813 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:40.152767    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:04:40.153397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:40.152824    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:40.153965 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:40.153941    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:04:41.157470 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:41.157417    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:04:45.156646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:45.156617    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:04:45.157240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:45.157211    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:04:46.162313 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:46.162282    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:04:46.162931 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:46.162901    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:04:55.157971 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:55.157935    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:04:56.163697 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:04:56.163659    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:05.157598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:05.157514    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:05:06.163550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:06.163509    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:15.157744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:15.157705    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:05:16.163797 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:16.163756    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:25.157902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:25.157862    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:05:26.163487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:26.163437    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:35.157546 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:35.157504    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:05:36.163305 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:36.163269    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:45.158044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:45.158002    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:05:46.163857 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:46.163810    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:46.723740 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:46.723697    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:05:55.157778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:55.157737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:05:56.729119 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:05:56.729082    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:06:03.119450 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.119406    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:06:03.120004 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.119787    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" containerID="cri-o://4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88" gracePeriod=30
Jul 29 11:06:03.120004 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.119969    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kube-rbac-proxy" containerID="cri-o://89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4" gracePeriod=30
Jul 29 11:06:03.157411 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.157386    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:06:03.157630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.157589    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" containerID="cri-o://23b062062cc02640c51f52e98cceeee7dde55376e1f12c4f4cc0265bcd72e9de" gracePeriod=30
Jul 29 11:06:03.157693 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.157639    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kube-rbac-proxy" containerID="cri-o://37d52f7650af6bd47652b058b9211af5b035fca0cd0f6a679d3d260f799ea5e8" gracePeriod=30
Jul 29 11:06:03.517006 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.516973    2551 generic.go:356] "Generic (PLEG): container finished" podID="26ce6cdf-0af2-4665-9fb2-337986959768" containerID="89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4" exitCode=2
Jul 29 11:06:03.517213 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.517042    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerDied","Data":"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"}
Jul 29 11:06:03.521831 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.521808    2551 generic.go:356] "Generic (PLEG): container finished" podID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerID="37d52f7650af6bd47652b058b9211af5b035fca0cd0f6a679d3d260f799ea5e8" exitCode=2
Jul 29 11:06:03.521956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:03.521878    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerDied","Data":"37d52f7650af6bd47652b058b9211af5b035fca0cd0f6a679d3d260f799ea5e8"}
Jul 29 11:06:05.153047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:05.153008    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 29 11:06:05.157295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:05.157271    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 29 11:06:06.158380 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:06.158340    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 29 11:06:06.723651 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:06.723611    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 29 11:06:07.479487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.479464    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:06:07.540017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.539938    2551 generic.go:356] "Generic (PLEG): container finished" podID="26ce6cdf-0af2-4665-9fb2-337986959768" containerID="4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88" exitCode=0
Jul 29 11:06:07.540017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.539991    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"
Jul 29 11:06:07.540229 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.540021    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerDied","Data":"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"}
Jul 29 11:06:07.540229 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.540090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9" event={"ID":"26ce6cdf-0af2-4665-9fb2-337986959768","Type":"ContainerDied","Data":"bea99c784586180f96999a132eec96189742da48ddabb3f6110677aff747da55"}
Jul 29 11:06:07.540229 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.540147    2551 scope.go:117] "RemoveContainer" containerID="89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"
Jul 29 11:06:07.543520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.543292    2551 generic.go:356] "Generic (PLEG): container finished" podID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerID="23b062062cc02640c51f52e98cceeee7dde55376e1f12c4f4cc0265bcd72e9de" exitCode=0
Jul 29 11:06:07.543520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.543330    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerDied","Data":"23b062062cc02640c51f52e98cceeee7dde55376e1f12c4f4cc0265bcd72e9de"}
Jul 29 11:06:07.549971 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.549950    2551 scope.go:117] "RemoveContainer" containerID="4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"
Jul 29 11:06:07.557220 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.557203    2551 scope.go:117] "RemoveContainer" containerID="e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"
Jul 29 11:06:07.564558 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.564529    2551 scope.go:117] "RemoveContainer" containerID="89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"
Jul 29 11:06:07.564823 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:06:07.564805    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4\": container with ID starting with 89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4 not found: ID does not exist" containerID="89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"
Jul 29 11:06:07.564880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.564838    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4"} err="failed to get container status \"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4\": rpc error: code = NotFound desc = could not find container \"89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4\": container with ID starting with 89e0c44265cbc0c01288146fdf1c5732796285793c1c3230e13fd5e7180cbbf4 not found: ID does not exist"
Jul 29 11:06:07.564880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.564862    2551 scope.go:117] "RemoveContainer" containerID="4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"
Jul 29 11:06:07.565116 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:06:07.565101    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88\": container with ID starting with 4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88 not found: ID does not exist" containerID="4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"
Jul 29 11:06:07.565214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.565120    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88"} err="failed to get container status \"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88\": rpc error: code = NotFound desc = could not find container \"4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88\": container with ID starting with 4ad9d9240f9e7d5a412b8dc279efe14f0ec20d516b1c65cc4ae78fb84264dd88 not found: ID does not exist"
Jul 29 11:06:07.565214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.565154    2551 scope.go:117] "RemoveContainer" containerID="e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"
Jul 29 11:06:07.565406 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:06:07.565385    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0\": container with ID starting with e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0 not found: ID does not exist" containerID="e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"
Jul 29 11:06:07.565449 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.565415    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0"} err="failed to get container status \"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0\": rpc error: code = NotFound desc = could not find container \"e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0\": container with ID starting with e3bbf5118aaca9ec6fce283898ab34d6f347e4ed814ec7fd1322fb6cb1d4e1d0 not found: ID does not exist"
Jul 29 11:06:07.600961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.600938    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:06:07.612097 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612077    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location\") pod \"26ce6cdf-0af2-4665-9fb2-337986959768\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") "
Jul 29 11:06:07.612193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612146    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") pod \"26ce6cdf-0af2-4665-9fb2-337986959768\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") "
Jul 29 11:06:07.612193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612173    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config\") pod \"26ce6cdf-0af2-4665-9fb2-337986959768\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") "
Jul 29 11:06:07.612307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612200    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert\") pod \"26ce6cdf-0af2-4665-9fb2-337986959768\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") "
Jul 29 11:06:07.612307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612225    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv8dv\" (UniqueName: \"kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv\") pod \"26ce6cdf-0af2-4665-9fb2-337986959768\" (UID: \"26ce6cdf-0af2-4665-9fb2-337986959768\") "
Jul 29 11:06:07.612459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612431    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26ce6cdf-0af2-4665-9fb2-337986959768" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:07.612553 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612531    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26ce6cdf-0af2-4665-9fb2-337986959768" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:06:07.612610 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.612536    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26ce6cdf-0af2-4665-9fb2-337986959768" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:06:07.614260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.614239    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv" (OuterVolumeSpecName: "kube-api-access-vv8dv") pod "26ce6cdf-0af2-4665-9fb2-337986959768" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768"). InnerVolumeSpecName "kube-api-access-vv8dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:06:07.614318 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.614288    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26ce6cdf-0af2-4665-9fb2-337986959768" (UID: "26ce6cdf-0af2-4665-9fb2-337986959768"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:06:07.712885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.712848    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config\") pod \"607ad23e-2acf-4a9d-b271-b21350c7460a\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") "
Jul 29 11:06:07.712885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.712892    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwllr\" (UniqueName: \"kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr\") pod \"607ad23e-2acf-4a9d-b271-b21350c7460a\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") "
Jul 29 11:06:07.713099 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.712934    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert\") pod \"607ad23e-2acf-4a9d-b271-b21350c7460a\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") "
Jul 29 11:06:07.713099 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.712982    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls\") pod \"607ad23e-2acf-4a9d-b271-b21350c7460a\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") "
Jul 29 11:06:07.713099 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713009    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location\") pod \"607ad23e-2acf-4a9d-b271-b21350c7460a\" (UID: \"607ad23e-2acf-4a9d-b271-b21350c7460a\") "
Jul 29 11:06:07.713279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713260    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26ce6cdf-0af2-4665-9fb2-337986959768-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.713330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713285    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26ce6cdf-0af2-4665-9fb2-337986959768-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.713330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713300    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.713330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713313    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26ce6cdf-0af2-4665-9fb2-337986959768-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.713330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713327    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8dv\" (UniqueName: \"kubernetes.io/projected/26ce6cdf-0af2-4665-9fb2-337986959768-kube-api-access-vv8dv\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.713518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713427    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "607ad23e-2acf-4a9d-b271-b21350c7460a" (UID: "607ad23e-2acf-4a9d-b271-b21350c7460a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:06:07.713518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713433    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "607ad23e-2acf-4a9d-b271-b21350c7460a" (UID: "607ad23e-2acf-4a9d-b271-b21350c7460a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:07.713518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.713422    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "607ad23e-2acf-4a9d-b271-b21350c7460a" (UID: "607ad23e-2acf-4a9d-b271-b21350c7460a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:06:07.715098 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.715079    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "607ad23e-2acf-4a9d-b271-b21350c7460a" (UID: "607ad23e-2acf-4a9d-b271-b21350c7460a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:06:07.715183 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.715146    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr" (OuterVolumeSpecName: "kube-api-access-hwllr") pod "607ad23e-2acf-4a9d-b271-b21350c7460a" (UID: "607ad23e-2acf-4a9d-b271-b21350c7460a"). InnerVolumeSpecName "kube-api-access-hwllr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:06:07.814439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.814361    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.814439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.814387    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/607ad23e-2acf-4a9d-b271-b21350c7460a-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.814439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.814396    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/607ad23e-2acf-4a9d-b271-b21350c7460a-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.814439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.814407    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.814439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.814417    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwllr\" (UniqueName: \"kubernetes.io/projected/607ad23e-2acf-4a9d-b271-b21350c7460a-kube-api-access-hwllr\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:06:07.862179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.862146    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:06:07.865826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:07.865802    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-75dfd479cd-26gh9"]
Jul 29 11:06:08.548960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.548875    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"
Jul 29 11:06:08.548960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.548886    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z" event={"ID":"607ad23e-2acf-4a9d-b271-b21350c7460a","Type":"ContainerDied","Data":"aa2debe57ef8f091cdc4a3c46fe537cf406f8e3b0473a55f2574670e4ebd9b93"}
Jul 29 11:06:08.549504 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.548991    2551 scope.go:117] "RemoveContainer" containerID="37d52f7650af6bd47652b058b9211af5b035fca0cd0f6a679d3d260f799ea5e8"
Jul 29 11:06:08.557461 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.557442    2551 scope.go:117] "RemoveContainer" containerID="23b062062cc02640c51f52e98cceeee7dde55376e1f12c4f4cc0265bcd72e9de"
Jul 29 11:06:08.564949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.564927    2551 scope.go:117] "RemoveContainer" containerID="5a458151681e997271d3599960b347cf47facfd1ff15c8f584fa2524740a7e4e"
Jul 29 11:06:08.570965 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.570945    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:06:08.574858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.574836    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-865fc6b775-zb88z"]
Jul 29 11:06:08.728595 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.728560    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" path="/var/lib/kubelet/pods/26ce6cdf-0af2-4665-9fb2-337986959768/volumes"
Jul 29 11:06:08.729057 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:08.729044    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" path="/var/lib/kubelet/pods/607ad23e-2acf-4a9d-b271-b21350c7460a/volumes"
Jul 29 11:06:13.418698 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.418667    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419001    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kube-rbac-proxy"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419013    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kube-rbac-proxy"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419024    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419030    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419039    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="storage-initializer"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419045    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="storage-initializer"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419057    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="storage-initializer"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419062    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="storage-initializer"
Jul 29 11:06:13.419067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419069    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kube-rbac-proxy"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419074    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kube-rbac-proxy"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419084    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419088    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419153    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kserve-container"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419160    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kserve-container"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419166    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ce6cdf-0af2-4665-9fb2-337986959768" containerName="kube-rbac-proxy"
Jul 29 11:06:13.419364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.419173    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="607ad23e-2acf-4a9d-b271-b21350c7460a" containerName="kube-rbac-proxy"
Jul 29 11:06:13.422265 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.422250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.424664 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.424634    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 29 11:06:13.424829 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.424694    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:06:13.424829 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.424748    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 29 11:06:13.424950 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.424905    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:06:13.425756 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.425735    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:06:13.425854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.425756    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:06:13.425854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.425761    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:06:13.432692 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.432669    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:06:13.556397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.556369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.556540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.556401    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.556540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.556422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.556540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.556490    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.556645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.556544    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gkb\" (UniqueName: \"kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.657778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.657753    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gkb\" (UniqueName: \"kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.657921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.657796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.657921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.657817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.657921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.657836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.657921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.657862    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.658298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.658278    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.658434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.658418    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.658509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.658493    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.660285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.660264    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.665323 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.665302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gkb\" (UniqueName: \"kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb\") pod \"isvc-lightgbm-predictor-54f85ddf88-h7blj\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.732108 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.732086    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:13.860605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:13.860503    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:06:13.862880 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:06:13.862854    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c17ce7_4ea2_4069_9b5f_430a2992a5b2.slice/crio-c20197041abbc38bf91bef5d972576ae47469da8152af23880cf259fc940e9f6 WatchSource:0}: Error finding container c20197041abbc38bf91bef5d972576ae47469da8152af23880cf259fc940e9f6: Status 404 returned error can't find the container with id c20197041abbc38bf91bef5d972576ae47469da8152af23880cf259fc940e9f6
Jul 29 11:06:14.575629 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:14.575593    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerStarted","Data":"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"}
Jul 29 11:06:14.576073 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:14.575661    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerStarted","Data":"c20197041abbc38bf91bef5d972576ae47469da8152af23880cf259fc940e9f6"}
Jul 29 11:06:17.589090 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:17.589062    2551 generic.go:356] "Generic (PLEG): container finished" podID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerID="339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57" exitCode=0
Jul 29 11:06:17.589459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:17.589143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerDied","Data":"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"}
Jul 29 11:06:17.590275 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:17.590261    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:06:24.623163 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:24.623110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerStarted","Data":"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"}
Jul 29 11:06:24.623546 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:24.623192    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerStarted","Data":"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"}
Jul 29 11:06:24.623546 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:24.623383    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:24.645953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:24.645912    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podStartSLOduration=1.6965816569999999 podStartE2EDuration="11.645898656s" totalImagesPullingTime="6.331068996s" totalInitContainerRuntime="3.618248003s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:06:24.644186787 +0000 UTC m=+2236.546415184" watchObservedRunningTime="2026-07-29 11:06:24.645898656 +0000 UTC m=+2236.548127052"
Jul 29 11:06:25.627299 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:25.627263    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:25.628527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:25.628500    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:06:26.631465 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:26.631425    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:06:31.637010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:31.636935    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:06:31.639339 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:31.637495    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:06:41.637592 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:41.637557    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:06:51.637854 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:06:51.637813    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:07:01.637439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:01.637402    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:07:11.637977 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:11.637930    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:07:21.638425 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:21.638388    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:07:31.637772 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:31.637734    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 29 11:07:41.638997 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:41.638960    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:07:43.570614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:43.570578    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:07:43.571000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:43.570792    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container" containerID="cri-o://af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f" gracePeriod=30
Jul 29 11:07:43.571000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:43.570870    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kube-rbac-proxy" containerID="cri-o://e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc" gracePeriod=30
Jul 29 11:07:43.948532 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:43.948464    2551 generic.go:356] "Generic (PLEG): container finished" podID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerID="e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc" exitCode=2
Jul 29 11:07:43.948532 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:43.948501    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerDied","Data":"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"}
Jul 29 11:07:46.631710 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:46.631662    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 29 11:07:47.416513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.416489    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:07:47.453851 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.453826    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location\") pod \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") "
Jul 29 11:07:47.453972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.453877    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls\") pod \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") "
Jul 29 11:07:47.454013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.453977    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert\") pod \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") "
Jul 29 11:07:47.454013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454005    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9gkb\" (UniqueName: \"kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb\") pod \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") "
Jul 29 11:07:47.454092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454027    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config\") pod \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\" (UID: \"77c17ce7-4ea2-4069-9b5f-430a2992a5b2\") "
Jul 29 11:07:47.454239 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454213    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77c17ce7-4ea2-4069-9b5f-430a2992a5b2" (UID: "77c17ce7-4ea2-4069-9b5f-430a2992a5b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:07:47.454376 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454357    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:07:47.454458 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454373    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77c17ce7-4ea2-4069-9b5f-430a2992a5b2" (UID: "77c17ce7-4ea2-4069-9b5f-430a2992a5b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:07:47.454458 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.454433    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77c17ce7-4ea2-4069-9b5f-430a2992a5b2" (UID: "77c17ce7-4ea2-4069-9b5f-430a2992a5b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:07:47.455880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.455863    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77c17ce7-4ea2-4069-9b5f-430a2992a5b2" (UID: "77c17ce7-4ea2-4069-9b5f-430a2992a5b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:07:47.455989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.455976    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb" (OuterVolumeSpecName: "kube-api-access-d9gkb") pod "77c17ce7-4ea2-4069-9b5f-430a2992a5b2" (UID: "77c17ce7-4ea2-4069-9b5f-430a2992a5b2"). InnerVolumeSpecName "kube-api-access-d9gkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:07:47.555468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.555439    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:07:47.555582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.555469    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9gkb\" (UniqueName: \"kubernetes.io/projected/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-api-access-d9gkb\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:07:47.555582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.555486    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:07:47.555582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.555500    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77c17ce7-4ea2-4069-9b5f-430a2992a5b2-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:07:47.973741 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.973711    2551 generic.go:356] "Generic (PLEG): container finished" podID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerID="af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f" exitCode=0
Jul 29 11:07:47.974193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.973783    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerDied","Data":"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"}
Jul 29 11:07:47.974193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.973818    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"
Jul 29 11:07:47.974193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.973887    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj" event={"ID":"77c17ce7-4ea2-4069-9b5f-430a2992a5b2","Type":"ContainerDied","Data":"c20197041abbc38bf91bef5d972576ae47469da8152af23880cf259fc940e9f6"}
Jul 29 11:07:47.974193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.973926    2551 scope.go:117] "RemoveContainer" containerID="e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"
Jul 29 11:07:47.982068 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.982048    2551 scope.go:117] "RemoveContainer" containerID="af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"
Jul 29 11:07:47.990249 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.990225    2551 scope.go:117] "RemoveContainer" containerID="339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"
Jul 29 11:07:47.997145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997103    2551 scope.go:117] "RemoveContainer" containerID="e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"
Jul 29 11:07:47.997412 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:07:47.997392    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc\": container with ID starting with e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc not found: ID does not exist" containerID="e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"
Jul 29 11:07:47.997497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997422    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc"} err="failed to get container status \"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc\": rpc error: code = NotFound desc = could not find container \"e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc\": container with ID starting with e3c007f62110fc6c343437a3983bb347b1b3861200d02c7b3e892658f87aadfc not found: ID does not exist"
Jul 29 11:07:47.997497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997445    2551 scope.go:117] "RemoveContainer" containerID="af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"
Jul 29 11:07:47.997720 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:07:47.997702    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f\": container with ID starting with af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f not found: ID does not exist" containerID="af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"
Jul 29 11:07:47.997759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997726    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f"} err="failed to get container status \"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f\": rpc error: code = NotFound desc = could not find container \"af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f\": container with ID starting with af2c0f2b39790c3ae0be1165d5b4359418c8482ae1851f772957b4f867ed420f not found: ID does not exist"
Jul 29 11:07:47.997759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997743    2551 scope.go:117] "RemoveContainer" containerID="339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"
Jul 29 11:07:47.997959 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:07:47.997940    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57\": container with ID starting with 339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57 not found: ID does not exist" containerID="339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"
Jul 29 11:07:47.998030 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:47.997966    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57"} err="failed to get container status \"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57\": rpc error: code = NotFound desc = could not find container \"339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57\": container with ID starting with 339053b31632f8c4add4c32032ba100abdfae49eb06f68d9fbe1fe92fa522a57 not found: ID does not exist"
Jul 29 11:07:48.014019 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.013997    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:07:48.016115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.016094    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-h7blj"]
Jul 29 11:07:48.728635 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.728605    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" path="/var/lib/kubelet/pods/77c17ce7-4ea2-4069-9b5f-430a2992a5b2/volumes"
Jul 29 11:07:48.734454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734428    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:07:48.734725 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734714    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kube-rbac-proxy"
Jul 29 11:07:48.734779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734728    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kube-rbac-proxy"
Jul 29 11:07:48.734779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734747    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container"
Jul 29 11:07:48.734779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734753    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container"
Jul 29 11:07:48.734779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734762    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="storage-initializer"
Jul 29 11:07:48.734779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734768    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="storage-initializer"
Jul 29 11:07:48.734933 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734817    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kserve-container"
Jul 29 11:07:48.734933 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.734825    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c17ce7-4ea2-4069-9b5f-430a2992a5b2" containerName="kube-rbac-proxy"
Jul 29 11:07:48.740428 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.740412    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.743121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.743104    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:07:48.743121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.743110    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:07:48.743280 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.743118    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:07:48.743977 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.743953    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:07:48.743977 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.743959    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:07:48.744166 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.744018    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:07:48.744166 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.744018    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 29 11:07:48.748022 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.748002    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:07:48.765803 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.765775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.765938 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.765807    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gkt\" (UniqueName: \"kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.766142 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.766097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.766254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.766173    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.766254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.766202    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.866602 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.866571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.866717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.866608    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gkt\" (UniqueName: \"kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.866717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.866666    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.866818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.866723    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.866818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.866766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.867247 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.867221    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.867343 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.867226    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.867404 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.867388    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.869178 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.869159    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:48.876405 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:48.876385    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gkt\" (UniqueName: \"kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:49.050711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:49.050630    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:49.176138 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:49.176099    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:07:49.178521 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:07:49.178489    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e5184b_22b6_46e7_b059_dd9a97b94f24.slice/crio-859ba837ff4bf59a23070fd1f7a1e6c72078783ad533cf69777e8a2434aa3e7c WatchSource:0}: Error finding container 859ba837ff4bf59a23070fd1f7a1e6c72078783ad533cf69777e8a2434aa3e7c: Status 404 returned error can't find the container with id 859ba837ff4bf59a23070fd1f7a1e6c72078783ad533cf69777e8a2434aa3e7c
Jul 29 11:07:49.984720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:49.984682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerStarted","Data":"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"}
Jul 29 11:07:49.984898 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:49.984748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerStarted","Data":"859ba837ff4bf59a23070fd1f7a1e6c72078783ad533cf69777e8a2434aa3e7c"}
Jul 29 11:07:54.004559 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:54.004520    2551 generic.go:356] "Generic (PLEG): container finished" podID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerID="806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1" exitCode=0
Jul 29 11:07:54.004960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:54.004571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerDied","Data":"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"}
Jul 29 11:07:55.011260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:55.011222    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerStarted","Data":"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"}
Jul 29 11:07:55.011732 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:55.011295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerStarted","Data":"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"}
Jul 29 11:07:55.011732 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:55.011522    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:55.030690 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:55.030637    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podStartSLOduration=3.110800943 podStartE2EDuration="7.030620102s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.919819159s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:07:55.029959923 +0000 UTC m=+2326.932188320" watchObservedRunningTime="2026-07-29 11:07:55.030620102 +0000 UTC m=+2326.932848500"
Jul 29 11:07:56.015526 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:56.015493    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:07:56.016748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:56.016720    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:07:57.019608 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:07:57.019568    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:02.024335 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:02.024257    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:08:02.024728 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:02.024703    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:12.025019 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:12.024976    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:22.025306 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:22.025264    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:32.024694 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:32.024657    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:42.024863 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:42.024823    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:08:52.025987 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:08:52.025938    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:09:02.025179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:02.025120    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:09:08.754678 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:08.754651    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:09:08.758785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:08.758754    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:09:12.025352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:12.025310    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:09:19.371727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:19.371690    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:09:19.372237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:19.371909    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" containerID="cri-o://9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989" gracePeriod=30
Jul 29 11:09:19.372237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:19.371944    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kube-rbac-proxy" containerID="cri-o://1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f" gracePeriod=30
Jul 29 11:09:20.376516 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:20.376467    2551 generic.go:356] "Generic (PLEG): container finished" podID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerID="1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f" exitCode=2
Jul 29 11:09:20.376891 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:20.376530    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerDied","Data":"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"}
Jul 29 11:09:22.019941 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:22.019894    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 29 11:09:22.024900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:22.024871    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 29 11:09:23.217631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.217611    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:09:23.256039 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.255984    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls\") pod \"29e5184b-22b6-46e7-b059-dd9a97b94f24\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") "
Jul 29 11:09:23.256039 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256028    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert\") pod \"29e5184b-22b6-46e7-b059-dd9a97b94f24\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") "
Jul 29 11:09:23.256243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256059    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location\") pod \"29e5184b-22b6-46e7-b059-dd9a97b94f24\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") "
Jul 29 11:09:23.256243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256094    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config\") pod \"29e5184b-22b6-46e7-b059-dd9a97b94f24\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") "
Jul 29 11:09:23.256243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256203    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6gkt\" (UniqueName: \"kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt\") pod \"29e5184b-22b6-46e7-b059-dd9a97b94f24\" (UID: \"29e5184b-22b6-46e7-b059-dd9a97b94f24\") "
Jul 29 11:09:23.256472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256436    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29e5184b-22b6-46e7-b059-dd9a97b94f24" (UID: "29e5184b-22b6-46e7-b059-dd9a97b94f24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:09:23.256531 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256473    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29e5184b-22b6-46e7-b059-dd9a97b94f24" (UID: "29e5184b-22b6-46e7-b059-dd9a97b94f24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:09:23.256568 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.256549    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29e5184b-22b6-46e7-b059-dd9a97b94f24" (UID: "29e5184b-22b6-46e7-b059-dd9a97b94f24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:09:23.257994 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.257965    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29e5184b-22b6-46e7-b059-dd9a97b94f24" (UID: "29e5184b-22b6-46e7-b059-dd9a97b94f24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:09:23.258172 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.258142    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt" (OuterVolumeSpecName: "kube-api-access-f6gkt") pod "29e5184b-22b6-46e7-b059-dd9a97b94f24" (UID: "29e5184b-22b6-46e7-b059-dd9a97b94f24"). InnerVolumeSpecName "kube-api-access-f6gkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:09:23.357619 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.357598    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gkt\" (UniqueName: \"kubernetes.io/projected/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-api-access-f6gkt\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:09:23.357703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.357639    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29e5184b-22b6-46e7-b059-dd9a97b94f24-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:09:23.357703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.357651    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:09:23.357703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.357659    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e5184b-22b6-46e7-b059-dd9a97b94f24-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:09:23.357703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.357667    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29e5184b-22b6-46e7-b059-dd9a97b94f24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:09:23.390749 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.390723    2551 generic.go:356] "Generic (PLEG): container finished" podID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerID="9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989" exitCode=0
Jul 29 11:09:23.390844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.390778    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerDied","Data":"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"}
Jul 29 11:09:23.390844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.390787    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"
Jul 29 11:09:23.390921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.390844    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4" event={"ID":"29e5184b-22b6-46e7-b059-dd9a97b94f24","Type":"ContainerDied","Data":"859ba837ff4bf59a23070fd1f7a1e6c72078783ad533cf69777e8a2434aa3e7c"}
Jul 29 11:09:23.390921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.390875    2551 scope.go:117] "RemoveContainer" containerID="1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"
Jul 29 11:09:23.399241 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.399216    2551 scope.go:117] "RemoveContainer" containerID="9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"
Jul 29 11:09:23.406682 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.406661    2551 scope.go:117] "RemoveContainer" containerID="806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"
Jul 29 11:09:23.413586 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.413563    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:09:23.414543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.414527    2551 scope.go:117] "RemoveContainer" containerID="1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"
Jul 29 11:09:23.414844 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:09:23.414819    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f\": container with ID starting with 1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f not found: ID does not exist" containerID="1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"
Jul 29 11:09:23.414970 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.414850    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f"} err="failed to get container status \"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f\": rpc error: code = NotFound desc = could not find container \"1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f\": container with ID starting with 1a757be205b6f45aeacff059aab70539bdcc40e66154c9611e8b52ea651a456f not found: ID does not exist"
Jul 29 11:09:23.414970 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.414876    2551 scope.go:117] "RemoveContainer" containerID="9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"
Jul 29 11:09:23.415300 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:09:23.415272    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989\": container with ID starting with 9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989 not found: ID does not exist" containerID="9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"
Jul 29 11:09:23.415418 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.415306    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989"} err="failed to get container status \"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989\": rpc error: code = NotFound desc = could not find container \"9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989\": container with ID starting with 9bf9d58b11a6fac088cf732e241079cdc243021f93826c66f2a7ca460422b989 not found: ID does not exist"
Jul 29 11:09:23.415418 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.415330    2551 scope.go:117] "RemoveContainer" containerID="806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"
Jul 29 11:09:23.415711 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:09:23.415688    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1\": container with ID starting with 806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1 not found: ID does not exist" containerID="806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"
Jul 29 11:09:23.415782 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.415723    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1"} err="failed to get container status \"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1\": rpc error: code = NotFound desc = could not find container \"806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1\": container with ID starting with 806146a5ba1da47395b9ec924ff2663ed1281967b9116553c06027d21d2b33e1 not found: ID does not exist"
Jul 29 11:09:23.417046 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:23.417023    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-2c7s4"]
Jul 29 11:09:24.519419 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519385    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519728    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="storage-initializer"
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519739    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="storage-initializer"
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519749    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container"
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519755    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container"
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519765    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kube-rbac-proxy"
Jul 29 11:09:24.519799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519771    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kube-rbac-proxy"
Jul 29 11:09:24.519999 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519830    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kube-rbac-proxy"
Jul 29 11:09:24.519999 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.519846    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" containerName="kserve-container"
Jul 29 11:09:24.524579 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.524562    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.527045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.527019    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:09:24.527177 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.527163    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:09:24.527282 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.527265    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:09:24.527354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.527332    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:09:24.528018 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.528002    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:09:24.528078 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.528035    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:09:24.528078 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.528060    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 29 11:09:24.533026 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.533007    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:09:24.566748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.566723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.566837 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.566752    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.566837 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.566773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtpw\" (UniqueName: \"kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.566916 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.566866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.566916 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.566898    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667546 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.667519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.667550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.667573    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtpw\" (UniqueName: \"kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.667620    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667730 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.667642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.667730 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:09:24.667676    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 29 11:09:24.667815 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:09:24.667743    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls podName:3afaa9d6-0233-4a55-8c1f-7c493a56b7af nodeName:}" failed. No retries permitted until 2026-07-29 11:09:25.167723274 +0000 UTC m=+2417.069951650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 29 11:09:24.668146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.668106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.668299 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.668281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.668353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.668339    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.676476 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.676455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtpw\" (UniqueName: \"kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:24.728136 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:24.728109    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e5184b-22b6-46e7-b059-dd9a97b94f24" path="/var/lib/kubelet/pods/29e5184b-22b6-46e7-b059-dd9a97b94f24/volumes"
Jul 29 11:09:25.172001 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:25.171971    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:25.174495 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:25.174471    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:25.434962 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:25.434906    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:09:25.561235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:25.561210    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:09:25.562512 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:09:25.562485    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afaa9d6_0233_4a55_8c1f_7c493a56b7af.slice/crio-91923a0add4d22c5453967c54f5cfed2defca14ce927728efbb032080a55cad5 WatchSource:0}: Error finding container 91923a0add4d22c5453967c54f5cfed2defca14ce927728efbb032080a55cad5: Status 404 returned error can't find the container with id 91923a0add4d22c5453967c54f5cfed2defca14ce927728efbb032080a55cad5
Jul 29 11:09:26.406325 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:26.406287    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerStarted","Data":"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"}
Jul 29 11:09:26.406487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:26.406348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerStarted","Data":"91923a0add4d22c5453967c54f5cfed2defca14ce927728efbb032080a55cad5"}
Jul 29 11:09:30.425249 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:30.425211    2551 generic.go:356] "Generic (PLEG): container finished" podID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerID="5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e" exitCode=0
Jul 29 11:09:30.425666 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:09:30.425296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerDied","Data":"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"}
Jul 29 11:11:36.048820 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:36.048745    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerStarted","Data":"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"}
Jul 29 11:11:36.049199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:36.048822    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerStarted","Data":"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"}
Jul 29 11:11:36.049199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:36.048979    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:11:36.078047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:36.078005    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" podStartSLOduration=2.771879991 podStartE2EDuration="2m12.077991083s" totalImagesPullingTime="2m5.323945983s" totalInitContainerRuntime="3.982165109s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:11:36.076202383 +0000 UTC m=+2547.978430779" watchObservedRunningTime="2026-07-29 11:11:36.077991083 +0000 UTC m=+2547.980219481"
Jul 29 11:11:37.053852 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:37.053812    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:11:43.062825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:11:43.062794    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:12:13.066870 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:13.066830    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:12:14.681763 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:14.681733    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:12:14.682137 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:14.681946    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kserve-container" containerID="cri-o://fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911" gracePeriod=30
Jul 29 11:12:14.682137 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:14.681989    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kube-rbac-proxy" containerID="cri-o://3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b" gracePeriod=30
Jul 29 11:12:15.207619 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.207587    2551 generic.go:356] "Generic (PLEG): container finished" podID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerID="3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b" exitCode=2
Jul 29 11:12:15.207812 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.207655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerDied","Data":"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"}
Jul 29 11:12:15.621287 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.621266    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:12:15.714085 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714018    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config\") pod \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") "
Jul 29 11:12:15.714085 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714061    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location\") pod \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") "
Jul 29 11:12:15.714085 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714079    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert\") pod \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") "
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714226    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrtpw\" (UniqueName: \"kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw\") pod \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") "
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714271    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") pod \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\" (UID: \"3afaa9d6-0233-4a55-8c1f-7c493a56b7af\") "
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714403    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3afaa9d6-0233-4a55-8c1f-7c493a56b7af" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714417    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3afaa9d6-0233-4a55-8c1f-7c493a56b7af" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714430    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3afaa9d6-0233-4a55-8c1f-7c493a56b7af" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714547    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714560    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:15.714570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.714570    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:15.716260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.716235    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw" (OuterVolumeSpecName: "kube-api-access-wrtpw") pod "3afaa9d6-0233-4a55-8c1f-7c493a56b7af" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af"). InnerVolumeSpecName "kube-api-access-wrtpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:12:15.716354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.716262    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3afaa9d6-0233-4a55-8c1f-7c493a56b7af" (UID: "3afaa9d6-0233-4a55-8c1f-7c493a56b7af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:12:15.815585 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.815554    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:15.815585 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:15.815583    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtpw\" (UniqueName: \"kubernetes.io/projected/3afaa9d6-0233-4a55-8c1f-7c493a56b7af-kube-api-access-wrtpw\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:16.213226 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.213197    2551 generic.go:356] "Generic (PLEG): container finished" podID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerID="fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911" exitCode=0
Jul 29 11:12:16.213336 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.213255    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"
Jul 29 11:12:16.213336 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.213289    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerDied","Data":"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"}
Jul 29 11:12:16.213420 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.213356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs" event={"ID":"3afaa9d6-0233-4a55-8c1f-7c493a56b7af","Type":"ContainerDied","Data":"91923a0add4d22c5453967c54f5cfed2defca14ce927728efbb032080a55cad5"}
Jul 29 11:12:16.213420 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.213396    2551 scope.go:117] "RemoveContainer" containerID="3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"
Jul 29 11:12:16.221838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.221821    2551 scope.go:117] "RemoveContainer" containerID="fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"
Jul 29 11:12:16.228879 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.228863    2551 scope.go:117] "RemoveContainer" containerID="5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"
Jul 29 11:12:16.234165 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.234144    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:12:16.236014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.235996    2551 scope.go:117] "RemoveContainer" containerID="3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"
Jul 29 11:12:16.236384 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:12:16.236356    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b\": container with ID starting with 3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b not found: ID does not exist" containerID="3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"
Jul 29 11:12:16.236552 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.236396    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b"} err="failed to get container status \"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b\": rpc error: code = NotFound desc = could not find container \"3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b\": container with ID starting with 3d3b0fd969446ed4bff21ef76453fa258cebcc9e124c24cb4b2087db6fd57b7b not found: ID does not exist"
Jul 29 11:12:16.236552 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.236420    2551 scope.go:117] "RemoveContainer" containerID="fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"
Jul 29 11:12:16.236923 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:12:16.236897    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911\": container with ID starting with fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911 not found: ID does not exist" containerID="fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"
Jul 29 11:12:16.236989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.236930    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911"} err="failed to get container status \"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911\": rpc error: code = NotFound desc = could not find container \"fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911\": container with ID starting with fad6b2fc37c9c48632d9595dda1ec2f16cdd52a68e1262ec9652ed075c5ad911 not found: ID does not exist"
Jul 29 11:12:16.236989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.236951    2551 scope.go:117] "RemoveContainer" containerID="5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"
Jul 29 11:12:16.237298 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:12:16.237270    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e\": container with ID starting with 5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e not found: ID does not exist" containerID="5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"
Jul 29 11:12:16.237379 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.237306    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e"} err="failed to get container status \"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e\": rpc error: code = NotFound desc = could not find container \"5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e\": container with ID starting with 5b426fe0596b12a212635b5a70811f9a5cd07ed51039fb7ba4780a6c8989ac8e not found: ID does not exist"
Jul 29 11:12:16.238382 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.238364    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-2pkrs"]
Jul 29 11:12:16.728669 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:16.728634    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" path="/var/lib/kubelet/pods/3afaa9d6-0233-4a55-8c1f-7c493a56b7af/volumes"
Jul 29 11:12:19.830490 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830462    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830765    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kube-rbac-proxy"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830777    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kube-rbac-proxy"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830788    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kserve-container"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830794    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kserve-container"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830810    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="storage-initializer"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830816    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="storage-initializer"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830872    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kserve-container"
Jul 29 11:12:19.830873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.830878    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3afaa9d6-0233-4a55-8c1f-7c493a56b7af" containerName="kube-rbac-proxy"
Jul 29 11:12:19.832977 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.832960    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:19.835494 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.835472    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 29 11:12:19.835636 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.835493    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 29 11:12:19.835636 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.835494    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:12:19.836787 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.836765    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:12:19.836912 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.836808    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:12:19.836912 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.836842    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:12:19.836912 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.836810    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:12:19.846673 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.846562    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:19.948715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.948688    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:19.948715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.948718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:19.948902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.948749    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:19.948902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.948840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclxv\" (UniqueName: \"kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:19.948975 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:19.948920    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.049698 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.049674    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclxv\" (UniqueName: \"kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.049828 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.049713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.049828 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.049738    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.049942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.049859    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.049942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.049924    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.050152 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.050112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.050439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.050420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.050543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.050527    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.052401 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.052385    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.058851 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.058826    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclxv\" (UniqueName: \"kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.144475 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.144423    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:20.273501 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:20.273480    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:20.275288 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:12:20.275261    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd083f262_7767_42e0_9657_b69475c98378.slice/crio-73c9ba5b8ac28b2d169ac6fa8af5e642277923822aa47f6ccc2cfd2bef45076c WatchSource:0}: Error finding container 73c9ba5b8ac28b2d169ac6fa8af5e642277923822aa47f6ccc2cfd2bef45076c: Status 404 returned error can't find the container with id 73c9ba5b8ac28b2d169ac6fa8af5e642277923822aa47f6ccc2cfd2bef45076c
Jul 29 11:12:21.236659 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:21.236628    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerStarted","Data":"c3498e662163e13a6bd32fbc4332b54a05a79ccbf6e07dcd85b818b081201584"}
Jul 29 11:12:21.237063 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:21.236683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerStarted","Data":"73c9ba5b8ac28b2d169ac6fa8af5e642277923822aa47f6ccc2cfd2bef45076c"}
Jul 29 11:12:25.255688 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:25.255658    2551 generic.go:356] "Generic (PLEG): container finished" podID="d083f262-7767-42e0-9657-b69475c98378" containerID="c3498e662163e13a6bd32fbc4332b54a05a79ccbf6e07dcd85b818b081201584" exitCode=0
Jul 29 11:12:25.256086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:25.255730    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerDied","Data":"c3498e662163e13a6bd32fbc4332b54a05a79ccbf6e07dcd85b818b081201584"}
Jul 29 11:12:26.266527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.266489    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerStarted","Data":"c621e0f70f3e97a1b7fcda1e35c8f0e4c02398393dbf83f6289fd0d222a3c461"}
Jul 29 11:12:26.266925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.266552    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerStarted","Data":"256e7853c3a249344e11ee3ac9de27904dd7b1e836f69b1e25d86824ef4fe9df"}
Jul 29 11:12:26.266925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.266834    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:26.267069 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.267005    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:26.268286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.268260    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 29 11:12:26.287765 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:26.287714    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" podStartSLOduration=3.146488599 podStartE2EDuration="7.28769685s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.141208251s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:12:26.284966101 +0000 UTC m=+2598.187194497" watchObservedRunningTime="2026-07-29 11:12:26.28769685 +0000 UTC m=+2598.189925248"
Jul 29 11:12:27.270980 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:27.270938    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 29 11:12:32.275514 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:32.275437    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:32.276274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:32.276253    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:39.878375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:39.878332    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:39.878833 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:39.878655    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container" containerID="cri-o://256e7853c3a249344e11ee3ac9de27904dd7b1e836f69b1e25d86824ef4fe9df" gracePeriod=30
Jul 29 11:12:39.878833 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:39.878708    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kube-rbac-proxy" containerID="cri-o://c621e0f70f3e97a1b7fcda1e35c8f0e4c02398393dbf83f6289fd0d222a3c461" gracePeriod=30
Jul 29 11:12:40.326140 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.326097    2551 generic.go:356] "Generic (PLEG): container finished" podID="d083f262-7767-42e0-9657-b69475c98378" containerID="c621e0f70f3e97a1b7fcda1e35c8f0e4c02398393dbf83f6289fd0d222a3c461" exitCode=2
Jul 29 11:12:40.326140 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.326119    2551 generic.go:356] "Generic (PLEG): container finished" podID="d083f262-7767-42e0-9657-b69475c98378" containerID="256e7853c3a249344e11ee3ac9de27904dd7b1e836f69b1e25d86824ef4fe9df" exitCode=0
Jul 29 11:12:40.326316 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.326198    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerDied","Data":"c621e0f70f3e97a1b7fcda1e35c8f0e4c02398393dbf83f6289fd0d222a3c461"}
Jul 29 11:12:40.326316 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.326247    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerDied","Data":"256e7853c3a249344e11ee3ac9de27904dd7b1e836f69b1e25d86824ef4fe9df"}
Jul 29 11:12:40.419554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.419532    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:40.610784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.610721    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert\") pod \"d083f262-7767-42e0-9657-b69475c98378\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") "
Jul 29 11:12:40.610784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.610756    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xclxv\" (UniqueName: \"kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv\") pod \"d083f262-7767-42e0-9657-b69475c98378\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") "
Jul 29 11:12:40.610784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.610783    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls\") pod \"d083f262-7767-42e0-9657-b69475c98378\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") "
Jul 29 11:12:40.611024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.610835    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location\") pod \"d083f262-7767-42e0-9657-b69475c98378\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") "
Jul 29 11:12:40.611024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.610862    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config\") pod \"d083f262-7767-42e0-9657-b69475c98378\" (UID: \"d083f262-7767-42e0-9657-b69475c98378\") "
Jul 29 11:12:40.611201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.611147    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d083f262-7767-42e0-9657-b69475c98378" (UID: "d083f262-7767-42e0-9657-b69475c98378"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:12:40.611266 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.611212    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d083f262-7767-42e0-9657-b69475c98378" (UID: "d083f262-7767-42e0-9657-b69475c98378"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:12:40.611309 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.611275    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d083f262-7767-42e0-9657-b69475c98378" (UID: "d083f262-7767-42e0-9657-b69475c98378"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:12:40.612711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.612689    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d083f262-7767-42e0-9657-b69475c98378" (UID: "d083f262-7767-42e0-9657-b69475c98378"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:12:40.612784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.612743    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv" (OuterVolumeSpecName: "kube-api-access-xclxv") pod "d083f262-7767-42e0-9657-b69475c98378" (UID: "d083f262-7767-42e0-9657-b69475c98378"). InnerVolumeSpecName "kube-api-access-xclxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:12:40.712269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.712245    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d083f262-7767-42e0-9657-b69475c98378-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:40.712269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.712266    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:40.712402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.712276    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d083f262-7767-42e0-9657-b69475c98378-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:40.712402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.712285    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xclxv\" (UniqueName: \"kubernetes.io/projected/d083f262-7767-42e0-9657-b69475c98378-kube-api-access-xclxv\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:40.712402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:40.712293    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d083f262-7767-42e0-9657-b69475c98378-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:12:41.332891 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.332868    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"
Jul 29 11:12:41.333253 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.332888    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt" event={"ID":"d083f262-7767-42e0-9657-b69475c98378","Type":"ContainerDied","Data":"73c9ba5b8ac28b2d169ac6fa8af5e642277923822aa47f6ccc2cfd2bef45076c"}
Jul 29 11:12:41.333253 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.332952    2551 scope.go:117] "RemoveContainer" containerID="c621e0f70f3e97a1b7fcda1e35c8f0e4c02398393dbf83f6289fd0d222a3c461"
Jul 29 11:12:41.341279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.341249    2551 scope.go:117] "RemoveContainer" containerID="256e7853c3a249344e11ee3ac9de27904dd7b1e836f69b1e25d86824ef4fe9df"
Jul 29 11:12:41.348457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.348436    2551 scope.go:117] "RemoveContainer" containerID="c3498e662163e13a6bd32fbc4332b54a05a79ccbf6e07dcd85b818b081201584"
Jul 29 11:12:41.352344 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.352320    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:41.357675 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:41.357652    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v2rkt"]
Jul 29 11:12:42.727869 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:42.727836    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d083f262-7767-42e0-9657-b69475c98378" path="/var/lib/kubelet/pods/d083f262-7767-42e0-9657-b69475c98378/volumes"
Jul 29 11:12:45.027203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027168    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027474    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kube-rbac-proxy"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027484    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kube-rbac-proxy"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027492    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027499    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027516    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="storage-initializer"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027521    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="storage-initializer"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027569    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kserve-container"
Jul 29 11:12:45.027654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.027579    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d083f262-7767-42e0-9657-b69475c98378" containerName="kube-rbac-proxy"
Jul 29 11:12:45.030574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.030557    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.032842 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.032816    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:12:45.032983 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.032864    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:12:45.032983 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.032884    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:12:45.032983 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.032886    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:12:45.033706 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.033682    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:12:45.033818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.033786    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:12:45.033891 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.033860    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 29 11:12:45.039440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.039418    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:12:45.145626 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.145600    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.145735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.145630    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.145735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.145657    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.145822 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.145728    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.145822 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.145762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl69c\" (UniqueName: \"kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246347    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl69c\" (UniqueName: \"kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246445    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246515    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.246924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.246904    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.247147 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.247106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.247196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.247162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.249017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.248997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.254989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.254969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl69c\" (UniqueName: \"kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.341393 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.341341    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:45.469822 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:45.469799    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:12:45.472019 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:12:45.471994    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a94c81_5b27_46ad_b4b0_21e1d53b0ac6.slice/crio-7b0bf349893947ff5926048d2c6a81793b4b09ccb8403acc31a4697ae0c50d32 WatchSource:0}: Error finding container 7b0bf349893947ff5926048d2c6a81793b4b09ccb8403acc31a4697ae0c50d32: Status 404 returned error can't find the container with id 7b0bf349893947ff5926048d2c6a81793b4b09ccb8403acc31a4697ae0c50d32
Jul 29 11:12:46.360366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:46.360333    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerStarted","Data":"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"}
Jul 29 11:12:46.360787 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:46.360399    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerStarted","Data":"7b0bf349893947ff5926048d2c6a81793b4b09ccb8403acc31a4697ae0c50d32"}
Jul 29 11:12:49.377384 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:49.377349    2551 generic.go:356] "Generic (PLEG): container finished" podID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerID="f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187" exitCode=0
Jul 29 11:12:49.377699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:49.377419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerDied","Data":"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"}
Jul 29 11:12:50.384627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:50.384593    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerStarted","Data":"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"}
Jul 29 11:12:50.385013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:50.384655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerStarted","Data":"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"}
Jul 29 11:12:50.385013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:50.384870    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:50.385106 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:50.385025    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:12:50.406965 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:50.406927    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" podStartSLOduration=1.581809173 podStartE2EDuration="5.406913871s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.825104698s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:12:50.405705737 +0000 UTC m=+2622.307934146" watchObservedRunningTime="2026-07-29 11:12:50.406913871 +0000 UTC m=+2622.309142268"
Jul 29 11:12:56.395980 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:12:56.395949    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:13:26.400083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:26.400050    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:13:35.164351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:35.164317    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:13:35.164846 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:35.164579    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kserve-container" containerID="cri-o://38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79" gracePeriod=30
Jul 29 11:13:35.164846 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:35.164583    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kube-rbac-proxy" containerID="cri-o://1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310" gracePeriod=30
Jul 29 11:13:35.578486 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:35.578457    2551 generic.go:356] "Generic (PLEG): container finished" podID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerID="1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310" exitCode=2
Jul 29 11:13:35.578655 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:35.578525    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerDied","Data":"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"}
Jul 29 11:13:36.246056 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.246035    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:13:36.324336 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324271    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl69c\" (UniqueName: \"kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c\") pod \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") "
Jul 29 11:13:36.324336 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324311    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config\") pod \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") "
Jul 29 11:13:36.324510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324340    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert\") pod \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") "
Jul 29 11:13:36.324510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324379    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls\") pod \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") "
Jul 29 11:13:36.324510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324402    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location\") pod \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\" (UID: \"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6\") "
Jul 29 11:13:36.324696 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324671    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" (UID: "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:13:36.324769 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324749    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" (UID: "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:13:36.324852 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.324834    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" (UID: "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:13:36.326449 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.326426    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c" (OuterVolumeSpecName: "kube-api-access-zl69c") pod "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" (UID: "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6"). InnerVolumeSpecName "kube-api-access-zl69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:13:36.326549 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.326508    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" (UID: "c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:13:36.425534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.425511    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:13:36.425534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.425532    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:13:36.425657 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.425541    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:13:36.425657 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.425549    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:13:36.425657 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.425558    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl69c\" (UniqueName: \"kubernetes.io/projected/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6-kube-api-access-zl69c\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:13:36.584734 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.584668    2551 generic.go:356] "Generic (PLEG): container finished" podID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerID="38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79" exitCode=0
Jul 29 11:13:36.584734 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.584728    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"
Jul 29 11:13:36.584867 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.584740    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerDied","Data":"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"}
Jul 29 11:13:36.584867 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.584787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl" event={"ID":"c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6","Type":"ContainerDied","Data":"7b0bf349893947ff5926048d2c6a81793b4b09ccb8403acc31a4697ae0c50d32"}
Jul 29 11:13:36.584867 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.584813    2551 scope.go:117] "RemoveContainer" containerID="1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"
Jul 29 11:13:36.593047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.593031    2551 scope.go:117] "RemoveContainer" containerID="38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"
Jul 29 11:13:36.600620 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.600591    2551 scope.go:117] "RemoveContainer" containerID="f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"
Jul 29 11:13:36.606789 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.606769    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:13:36.607714 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.607700    2551 scope.go:117] "RemoveContainer" containerID="1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"
Jul 29 11:13:36.608011 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:13:36.607955    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310\": container with ID starting with 1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310 not found: ID does not exist" containerID="1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"
Jul 29 11:13:36.608011 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.607995    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310"} err="failed to get container status \"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310\": rpc error: code = NotFound desc = could not find container \"1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310\": container with ID starting with 1d2616ace4383d3a479cfca0d608430645d9271fb259a1e9849077ba38a71310 not found: ID does not exist"
Jul 29 11:13:36.608107 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.608018    2551 scope.go:117] "RemoveContainer" containerID="38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"
Jul 29 11:13:36.608368 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:13:36.608339    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79\": container with ID starting with 38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79 not found: ID does not exist" containerID="38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"
Jul 29 11:13:36.608486 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.608375    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79"} err="failed to get container status \"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79\": rpc error: code = NotFound desc = could not find container \"38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79\": container with ID starting with 38d8cee1ca15f95826a45aa7a94d0dd30447ae521525e65cf79f204c60aa4e79 not found: ID does not exist"
Jul 29 11:13:36.608486 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.608400    2551 scope.go:117] "RemoveContainer" containerID="f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"
Jul 29 11:13:36.608870 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:13:36.608849    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187\": container with ID starting with f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187 not found: ID does not exist" containerID="f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"
Jul 29 11:13:36.608957 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.608878    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187"} err="failed to get container status \"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187\": rpc error: code = NotFound desc = could not find container \"f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187\": container with ID starting with f3237f065d2209f7c8c5fa0f60e6672e57c403946cd430da1320cce29edae187 not found: ID does not exist"
Jul 29 11:13:36.610472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.610453    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sq6cl"]
Jul 29 11:13:36.728240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:36.728218    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" path="/var/lib/kubelet/pods/c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6/volumes"
Jul 29 11:13:40.318724 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.318692    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:13:40.319086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319056    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kube-rbac-proxy"
Jul 29 11:13:40.319086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319069    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kube-rbac-proxy"
Jul 29 11:13:40.319086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319080    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kserve-container"
Jul 29 11:13:40.319086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319085    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kserve-container"
Jul 29 11:13:40.319261 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319101    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="storage-initializer"
Jul 29 11:13:40.319261 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319107    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="storage-initializer"
Jul 29 11:13:40.319261 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319180    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kube-rbac-proxy"
Jul 29 11:13:40.319261 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.319189    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a94c81-5b27-46ad-b4b0-21e1d53b0ac6" containerName="kserve-container"
Jul 29 11:13:40.322297 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.322281    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.324936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.324905    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:13:40.324936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.324932    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 29 11:13:40.325112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.324941    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:13:40.325112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.324973    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:13:40.325112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.324944    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 29 11:13:40.325112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.325025    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:13:40.325292 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.325249    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:13:40.333094 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.333076    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:13:40.451512 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.451488    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.451605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.451521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ls94\" (UniqueName: \"kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.451605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.451555    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.451681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.451610    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.451681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.451641    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.552570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.552738 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.552738 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552598    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.552738 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:13:40.552687    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 29 11:13:40.552738 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552711    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.552925 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:13:40.552743    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls podName:704a359f-1542-4882-8553-2336931a7fa4 nodeName:}" failed. No retries permitted until 2026-07-29 11:13:41.052723638 +0000 UTC m=+2672.954952027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls") pod "isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" (UID: "704a359f-1542-4882-8553-2336931a7fa4") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 29 11:13:40.552925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552782    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ls94\" (UniqueName: \"kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.553013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.552976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.553341 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.553324    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.553520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.553498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:40.560785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:40.560766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ls94\" (UniqueName: \"kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:41.058000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.057972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:41.060282 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.060254    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6c966795cc-nmhfs\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:41.233447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.233421    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:41.360849 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.360821    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:13:41.363092 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:13:41.363063    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704a359f_1542_4882_8553_2336931a7fa4.slice/crio-e0802407487a491c79211eb141b0054faae97b1b2453de0285394befdb65dfa2 WatchSource:0}: Error finding container e0802407487a491c79211eb141b0054faae97b1b2453de0285394befdb65dfa2: Status 404 returned error can't find the container with id e0802407487a491c79211eb141b0054faae97b1b2453de0285394befdb65dfa2
Jul 29 11:13:41.609319 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.609245    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerStarted","Data":"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"}
Jul 29 11:13:41.609319 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:41.609310    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerStarted","Data":"e0802407487a491c79211eb141b0054faae97b1b2453de0285394befdb65dfa2"}
Jul 29 11:13:45.629218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:45.629184    2551 generic.go:356] "Generic (PLEG): container finished" podID="704a359f-1542-4882-8553-2336931a7fa4" containerID="b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7" exitCode=0
Jul 29 11:13:45.629594 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:45.629249    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerDied","Data":"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"}
Jul 29 11:13:45.683397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:45.683380    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:13:46.639078 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:46.638995    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerStarted","Data":"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"}
Jul 29 11:13:48.650477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.650445    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerStarted","Data":"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"}
Jul 29 11:13:48.650771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.650517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerStarted","Data":"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"}
Jul 29 11:13:48.650771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.650558    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:48.650771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.650584    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:48.650771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.650623    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:13:48.672439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:48.672399    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podStartSLOduration=2.270983733 podStartE2EDuration="8.672383574s" totalImagesPullingTime="2.810246023s" totalInitContainerRuntime="3.591153818s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:13:48.670777484 +0000 UTC m=+2680.573005881" watchObservedRunningTime="2026-07-29 11:13:48.672383574 +0000 UTC m=+2680.574611971"
Jul 29 11:13:54.660024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:13:54.659998    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:14:08.784448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:14:08.784374    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:14:08.790942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:14:08.790923    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:14:24.662528 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:14:24.662500    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:14:54.663725 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:14:54.663697    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:15:00.408168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.408119    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:15:00.408639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.408430    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container" containerID="cri-o://03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540" gracePeriod=30
Jul 29 11:15:00.408639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.408473    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-agent" containerID="cri-o://0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c" gracePeriod=30
Jul 29 11:15:00.408639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.408462    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" containerID="cri-o://850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa" gracePeriod=30
Jul 29 11:15:00.945380 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.945348    2551 generic.go:356] "Generic (PLEG): container finished" podID="704a359f-1542-4882-8553-2336931a7fa4" containerID="850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa" exitCode=2
Jul 29 11:15:00.945550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:00.945423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerDied","Data":"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"}
Jul 29 11:15:02.956406 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:02.956374    2551 generic.go:356] "Generic (PLEG): container finished" podID="704a359f-1542-4882-8553-2336931a7fa4" containerID="03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540" exitCode=0
Jul 29 11:15:02.956406 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:02.956412    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerDied","Data":"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"}
Jul 29 11:15:04.654803 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:04.654766    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:04.661266 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:04.661241    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 29 11:15:05.573390 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.573354    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:15:05.576991 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.576974    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.579665 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.579644    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 29 11:15:05.579784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.579652    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 29 11:15:05.587202 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.587180    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:15:05.662393 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.662368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmmq\" (UniqueName: \"kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.662707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.662403    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.662707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.662422    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.662707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.662521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.662707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.662586    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.763852 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.763819    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.763862    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.763941    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.764000    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764192 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.764047    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmmq\" (UniqueName: \"kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764429 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.764396    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764665 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.764644    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.764739 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.764662    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.766415 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.766400    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.772025 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.772003    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmmq\" (UniqueName: \"kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq\") pod \"isvc-paddle-predictor-6d5c9fd4f6-b74ld\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:05.888083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:05.888022    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:06.014915 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:06.014890    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:15:06.017427 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:15:06.017394    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7e9067_d7b5_486b_981f_7f4bd99d76f2.slice/crio-5f3a29fc1f9dfa5c79ae3c56a21423f85b3d49461f1582fe8b17dca946314472 WatchSource:0}: Error finding container 5f3a29fc1f9dfa5c79ae3c56a21423f85b3d49461f1582fe8b17dca946314472: Status 404 returned error can't find the container with id 5f3a29fc1f9dfa5c79ae3c56a21423f85b3d49461f1582fe8b17dca946314472
Jul 29 11:15:06.976098 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:06.976067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerStarted","Data":"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"}
Jul 29 11:15:06.976494 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:06.976120    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerStarted","Data":"5f3a29fc1f9dfa5c79ae3c56a21423f85b3d49461f1582fe8b17dca946314472"}
Jul 29 11:15:09.655643 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:09.655598    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:10.994083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:10.994051    2551 generic.go:356] "Generic (PLEG): container finished" podID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerID="332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202" exitCode=0
Jul 29 11:15:10.994476 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:10.994120    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerDied","Data":"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"}
Jul 29 11:15:14.655506 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:14.655458    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:14.655931 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:14.655617    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:15:14.660635 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:14.660602    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 29 11:15:19.655965 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:19.655913    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:23.049837 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.049805    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerStarted","Data":"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"}
Jul 29 11:15:23.050285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.049863    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerStarted","Data":"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"}
Jul 29 11:15:23.050285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.050152    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:23.050467 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.050310    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:23.051216 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.051198    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:15:23.068308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:23.068258    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podStartSLOduration=1.7311198540000001 podStartE2EDuration="18.068247208s" totalImagesPullingTime="11.661008192s" totalInitContainerRuntime="4.676119162s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:15:23.068189913 +0000 UTC m=+2774.970418310" watchObservedRunningTime="2026-07-29 11:15:23.068247208 +0000 UTC m=+2774.970475605"
Jul 29 11:15:24.054536 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:24.054496    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:15:24.654985 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:24.654938    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:24.660472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:24.660448    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 29 11:15:24.660564 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:24.660551    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:15:29.058212 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:29.058183    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:15:29.058733 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:29.058707    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:15:29.655437 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:29.655351    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 29 11:15:30.592813 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.592785    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:15:30.663014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.662991    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location\") pod \"704a359f-1542-4882-8553-2336931a7fa4\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") "
Jul 29 11:15:30.663196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663024    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert\") pod \"704a359f-1542-4882-8553-2336931a7fa4\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") "
Jul 29 11:15:30.663196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663115    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") pod \"704a359f-1542-4882-8553-2336931a7fa4\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") "
Jul 29 11:15:30.663196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663172    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ls94\" (UniqueName: \"kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94\") pod \"704a359f-1542-4882-8553-2336931a7fa4\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") "
Jul 29 11:15:30.663393 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663211    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config\") pod \"704a359f-1542-4882-8553-2336931a7fa4\" (UID: \"704a359f-1542-4882-8553-2336931a7fa4\") "
Jul 29 11:15:30.663393 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663372    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "704a359f-1542-4882-8553-2336931a7fa4" (UID: "704a359f-1542-4882-8553-2336931a7fa4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:15:30.663486 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663475    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/704a359f-1542-4882-8553-2336931a7fa4-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:15:30.663582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "704a359f-1542-4882-8553-2336931a7fa4" (UID: "704a359f-1542-4882-8553-2336931a7fa4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:15:30.663656 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.663633    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "704a359f-1542-4882-8553-2336931a7fa4" (UID: "704a359f-1542-4882-8553-2336931a7fa4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:15:30.665184 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.665165    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94" (OuterVolumeSpecName: "kube-api-access-2ls94") pod "704a359f-1542-4882-8553-2336931a7fa4" (UID: "704a359f-1542-4882-8553-2336931a7fa4"). InnerVolumeSpecName "kube-api-access-2ls94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:15:30.665273 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.665258    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "704a359f-1542-4882-8553-2336931a7fa4" (UID: "704a359f-1542-4882-8553-2336931a7fa4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:15:30.764179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.764149    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:15:30.764179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.764175    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/704a359f-1542-4882-8553-2336931a7fa4-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:15:30.764298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.764186    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ls94\" (UniqueName: \"kubernetes.io/projected/704a359f-1542-4882-8553-2336931a7fa4-kube-api-access-2ls94\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:15:30.764298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:30.764198    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/704a359f-1542-4882-8553-2336931a7fa4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:15:31.085222 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.085186    2551 generic.go:356] "Generic (PLEG): container finished" podID="704a359f-1542-4882-8553-2336931a7fa4" containerID="0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c" exitCode=137
Jul 29 11:15:31.085403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.085256    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"
Jul 29 11:15:31.085403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.085268    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerDied","Data":"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"}
Jul 29 11:15:31.085403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.085340    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs" event={"ID":"704a359f-1542-4882-8553-2336931a7fa4","Type":"ContainerDied","Data":"e0802407487a491c79211eb141b0054faae97b1b2453de0285394befdb65dfa2"}
Jul 29 11:15:31.085403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.085372    2551 scope.go:117] "RemoveContainer" containerID="850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"
Jul 29 11:15:31.095250 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.095153    2551 scope.go:117] "RemoveContainer" containerID="0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"
Jul 29 11:15:31.102893 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.102360    2551 scope.go:117] "RemoveContainer" containerID="03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"
Jul 29 11:15:31.104464 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.104437    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:15:31.108123 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.108101    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6c966795cc-nmhfs"]
Jul 29 11:15:31.110081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.110066    2551 scope.go:117] "RemoveContainer" containerID="b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"
Jul 29 11:15:31.116889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.116873    2551 scope.go:117] "RemoveContainer" containerID="850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"
Jul 29 11:15:31.117171 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:15:31.117151    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa\": container with ID starting with 850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa not found: ID does not exist" containerID="850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"
Jul 29 11:15:31.117264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117175    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa"} err="failed to get container status \"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa\": rpc error: code = NotFound desc = could not find container \"850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa\": container with ID starting with 850af71f93512c6816225d19db3d981ed9c4ac10ce838ba5bf13b688331466fa not found: ID does not exist"
Jul 29 11:15:31.117264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117190    2551 scope.go:117] "RemoveContainer" containerID="0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"
Jul 29 11:15:31.117425 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:15:31.117411    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c\": container with ID starting with 0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c not found: ID does not exist" containerID="0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"
Jul 29 11:15:31.117469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117429    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c"} err="failed to get container status \"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c\": rpc error: code = NotFound desc = could not find container \"0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c\": container with ID starting with 0dded9e6d2a2cd53fab0f0e68b7f015fc8549d87482a360b3fe226fedc00b11c not found: ID does not exist"
Jul 29 11:15:31.117469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117442    2551 scope.go:117] "RemoveContainer" containerID="03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"
Jul 29 11:15:31.117649 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:15:31.117634    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540\": container with ID starting with 03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540 not found: ID does not exist" containerID="03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"
Jul 29 11:15:31.117691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117654    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540"} err="failed to get container status \"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540\": rpc error: code = NotFound desc = could not find container \"03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540\": container with ID starting with 03c37a8215b186e2d1e2012ac1f42349068ca28af562b4995607ffdde7458540 not found: ID does not exist"
Jul 29 11:15:31.117691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117666    2551 scope.go:117] "RemoveContainer" containerID="b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"
Jul 29 11:15:31.117903 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:15:31.117885    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7\": container with ID starting with b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7 not found: ID does not exist" containerID="b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"
Jul 29 11:15:31.117948 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:31.117909    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7"} err="failed to get container status \"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7\": rpc error: code = NotFound desc = could not find container \"b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7\": container with ID starting with b6e212f4b049759647bbca1d926ecb878021543a5c38f8d4a4cbb66baf5fa3e7 not found: ID does not exist"
Jul 29 11:15:32.728935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:32.728903    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704a359f-1542-4882-8553-2336931a7fa4" path="/var/lib/kubelet/pods/704a359f-1542-4882-8553-2336931a7fa4/volumes"
Jul 29 11:15:39.059566 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:39.059522    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:15:49.059121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:49.059084    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:15:59.059420 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:15:59.059378    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:16:09.060047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:09.060020    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:16:17.202180 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:17.202120    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:16:17.202633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:17.202391    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" containerID="cri-o://0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199" gracePeriod=30
Jul 29 11:16:17.202633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:17.202432    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kube-rbac-proxy" containerID="cri-o://be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375" gracePeriod=30
Jul 29 11:16:18.290081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:18.290051    2551 generic.go:356] "Generic (PLEG): container finished" podID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerID="be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375" exitCode=2
Jul 29 11:16:18.290465 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:18.290123    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerDied","Data":"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"}
Jul 29 11:16:19.055702 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.055648    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 29 11:16:19.058934 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.058907    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 29 11:16:19.538076 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.538050    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:16:19.617282 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617255    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls\") pod \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") "
Jul 29 11:16:19.617398 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617296    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config\") pod \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") "
Jul 29 11:16:19.617398 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617326    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location\") pod \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") "
Jul 29 11:16:19.617398 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617355    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert\") pod \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") "
Jul 29 11:16:19.617398 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617392    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpmmq\" (UniqueName: \"kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq\") pod \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\" (UID: \"1d7e9067-d7b5-486b-981f-7f4bd99d76f2\") "
Jul 29 11:16:19.617743 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617712    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d7e9067-d7b5-486b-981f-7f4bd99d76f2" (UID: "1d7e9067-d7b5-486b-981f-7f4bd99d76f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:16:19.617743 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.617708    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d7e9067-d7b5-486b-981f-7f4bd99d76f2" (UID: "1d7e9067-d7b5-486b-981f-7f4bd99d76f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:16:19.619237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.619206    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d7e9067-d7b5-486b-981f-7f4bd99d76f2" (UID: "1d7e9067-d7b5-486b-981f-7f4bd99d76f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:16:19.619403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.619385    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq" (OuterVolumeSpecName: "kube-api-access-cpmmq") pod "1d7e9067-d7b5-486b-981f-7f4bd99d76f2" (UID: "1d7e9067-d7b5-486b-981f-7f4bd99d76f2"). InnerVolumeSpecName "kube-api-access-cpmmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:16:19.628186 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.628164    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d7e9067-d7b5-486b-981f-7f4bd99d76f2" (UID: "1d7e9067-d7b5-486b-981f-7f4bd99d76f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:16:19.717916 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.717894    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmmq\" (UniqueName: \"kubernetes.io/projected/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-api-access-cpmmq\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:16:19.717916 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.717916    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:16:19.718040 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.717926    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:16:19.718040 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.717934    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:16:19.718040 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:19.717942    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d7e9067-d7b5-486b-981f-7f4bd99d76f2-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:16:20.301245 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.301214    2551 generic.go:356] "Generic (PLEG): container finished" podID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerID="0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199" exitCode=0
Jul 29 11:16:20.301407 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.301309    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"
Jul 29 11:16:20.301407 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.301309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerDied","Data":"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"}
Jul 29 11:16:20.301407 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.301389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld" event={"ID":"1d7e9067-d7b5-486b-981f-7f4bd99d76f2","Type":"ContainerDied","Data":"5f3a29fc1f9dfa5c79ae3c56a21423f85b3d49461f1582fe8b17dca946314472"}
Jul 29 11:16:20.301522 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.301431    2551 scope.go:117] "RemoveContainer" containerID="be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"
Jul 29 11:16:20.311323 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.311304    2551 scope.go:117] "RemoveContainer" containerID="0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"
Jul 29 11:16:20.319210 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.319191    2551 scope.go:117] "RemoveContainer" containerID="332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"
Jul 29 11:16:20.325102 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.325077    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:16:20.328067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328043    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-b74ld"]
Jul 29 11:16:20.328333 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328310    2551 scope.go:117] "RemoveContainer" containerID="be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"
Jul 29 11:16:20.328600 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:16:20.328580    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375\": container with ID starting with be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375 not found: ID does not exist" containerID="be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"
Jul 29 11:16:20.328672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328609    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375"} err="failed to get container status \"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375\": rpc error: code = NotFound desc = could not find container \"be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375\": container with ID starting with be014f7c2a3624ca55766d8b3b15d7b73c317235e95182fc68ee62a96957f375 not found: ID does not exist"
Jul 29 11:16:20.328672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328628    2551 scope.go:117] "RemoveContainer" containerID="0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"
Jul 29 11:16:20.328855 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:16:20.328837    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199\": container with ID starting with 0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199 not found: ID does not exist" containerID="0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"
Jul 29 11:16:20.328905 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328861    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199"} err="failed to get container status \"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199\": rpc error: code = NotFound desc = could not find container \"0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199\": container with ID starting with 0299107dfdf5a36d369e5f0b19a503bfd01cc17d4507db583b5779b549d4e199 not found: ID does not exist"
Jul 29 11:16:20.328905 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.328877    2551 scope.go:117] "RemoveContainer" containerID="332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"
Jul 29 11:16:20.329101 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:16:20.329083    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202\": container with ID starting with 332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202 not found: ID does not exist" containerID="332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"
Jul 29 11:16:20.329179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.329106    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202"} err="failed to get container status \"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202\": rpc error: code = NotFound desc = could not find container \"332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202\": container with ID starting with 332ade47227fdd64d8a58541733b2f0b4d95d7d87bcceabf14bad40240ec4202 not found: ID does not exist"
Jul 29 11:16:20.728703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:20.728674    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" path="/var/lib/kubelet/pods/1d7e9067-d7b5-486b-981f-7f4bd99d76f2/volumes"
Jul 29 11:16:22.307570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307538    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307869    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="storage-initializer"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307881    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="storage-initializer"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307895    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="storage-initializer"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307900    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="storage-initializer"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307910    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307916    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307923    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-agent"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307940    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-agent"
Jul 29 11:16:22.307942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307948    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kube-rbac-proxy"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307953    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kube-rbac-proxy"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307960    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307966    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307974    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.307979    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.308028    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kube-rbac-proxy"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.308037    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7e9067-d7b5-486b-981f-7f4bd99d76f2" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.308044    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-agent"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.308050    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kserve-container"
Jul 29 11:16:22.308279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.308055    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="704a359f-1542-4882-8553-2336931a7fa4" containerName="kube-rbac-proxy"
Jul 29 11:16:22.311210 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.311190    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.313909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313878    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 29 11:16:22.313909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313903    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:16:22.314267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313940    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:16:22.314267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313889    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:16:22.314267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313878    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:16:22.314267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.313994    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:16:22.314763 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.314745    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:16:22.320394 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.320368    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:16:22.337771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.337746    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.337940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.337786    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.337940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.337826    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.337940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.337886    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.337940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.337930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h825p\" (UniqueName: \"kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438245 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438218    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438358 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438358 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438358 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438321    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438525 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h825p\" (UniqueName: \"kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.438525 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:16:22.438368    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 29 11:16:22.438525 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:16:22.438503    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls podName:fc97e533-efaa-416e-890c-d16bc9311ce5 nodeName:}" failed. No retries permitted until 2026-07-29 11:16:22.938477062 +0000 UTC m=+2834.840705442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-p9wcs" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 29 11:16:22.438696 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.438658    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.439031 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.439010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.439102 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.439008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.446718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.446695    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h825p\" (UniqueName: \"kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.941961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.941930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:22.944161 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:22.944139    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-p9wcs\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:23.222771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:23.222746    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:23.349820 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:23.349795    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:16:23.351863 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:16:23.351830    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc97e533_efaa_416e_890c_d16bc9311ce5.slice/crio-8fdcec6ae77cbeab5ef1f3ecf8592b762644cd2686d4670258e6fc7440d44bf1 WatchSource:0}: Error finding container 8fdcec6ae77cbeab5ef1f3ecf8592b762644cd2686d4670258e6fc7440d44bf1: Status 404 returned error can't find the container with id 8fdcec6ae77cbeab5ef1f3ecf8592b762644cd2686d4670258e6fc7440d44bf1
Jul 29 11:16:24.323199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:24.323159    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerStarted","Data":"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"}
Jul 29 11:16:24.323391 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:24.323226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerStarted","Data":"8fdcec6ae77cbeab5ef1f3ecf8592b762644cd2686d4670258e6fc7440d44bf1"}
Jul 29 11:16:28.341639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:28.341602    2551 generic.go:356] "Generic (PLEG): container finished" podID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerID="2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc" exitCode=0
Jul 29 11:16:28.342028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:28.341676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerDied","Data":"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"}
Jul 29 11:16:29.347528 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:29.347494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerStarted","Data":"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"}
Jul 29 11:16:29.348016 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:29.347557    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerStarted","Data":"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"}
Jul 29 11:16:29.348016 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:29.347780    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:29.368896 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:29.368855    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podStartSLOduration=3.018250224 podStartE2EDuration="7.368840889s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.350590665s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:16:29.366415329 +0000 UTC m=+2841.268643764" watchObservedRunningTime="2026-07-29 11:16:29.368840889 +0000 UTC m=+2841.271069286"
Jul 29 11:16:30.352347 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:30.352316    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:30.353509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:30.353480    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:16:31.356706 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:31.356664    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:16:36.362784 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:36.362757    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:16:36.365052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:36.363274    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:16:46.363574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:46.363532    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:16:56.363215 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:16:56.363182    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:17:06.363545 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:06.363449    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 29 11:17:16.364376 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:16.364341    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:17:23.865631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:23.865598    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:17:23.866068 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:23.865858    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container" containerID="cri-o://e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a" gracePeriod=30
Jul 29 11:17:23.866068 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:23.865916    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kube-rbac-proxy" containerID="cri-o://5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db" gracePeriod=30
Jul 29 11:17:24.599186 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:24.599153    2551 generic.go:356] "Generic (PLEG): container finished" podID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerID="5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db" exitCode=2
Jul 29 11:17:24.599371 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:24.599221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerDied","Data":"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"}
Jul 29 11:17:26.311472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.311439    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:17:26.331287 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331258    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") pod \"fc97e533-efaa-416e-890c-d16bc9311ce5\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") "
Jul 29 11:17:26.331426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331293    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert\") pod \"fc97e533-efaa-416e-890c-d16bc9311ce5\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") "
Jul 29 11:17:26.331426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config\") pod \"fc97e533-efaa-416e-890c-d16bc9311ce5\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") "
Jul 29 11:17:26.331426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331334    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h825p\" (UniqueName: \"kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p\") pod \"fc97e533-efaa-416e-890c-d16bc9311ce5\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") "
Jul 29 11:17:26.331426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331400    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location\") pod \"fc97e533-efaa-416e-890c-d16bc9311ce5\" (UID: \"fc97e533-efaa-416e-890c-d16bc9311ce5\") "
Jul 29 11:17:26.331703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc97e533-efaa-416e-890c-d16bc9311ce5" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:17:26.331826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.331744    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc97e533-efaa-416e-890c-d16bc9311ce5" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:17:26.333619 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.333580    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p" (OuterVolumeSpecName: "kube-api-access-h825p") pod "fc97e533-efaa-416e-890c-d16bc9311ce5" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5"). InnerVolumeSpecName "kube-api-access-h825p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:17:26.333726 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.333677    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc97e533-efaa-416e-890c-d16bc9311ce5" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:17:26.341665 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.341641    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc97e533-efaa-416e-890c-d16bc9311ce5" (UID: "fc97e533-efaa-416e-890c-d16bc9311ce5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:17:26.431972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.431943    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc97e533-efaa-416e-890c-d16bc9311ce5-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:17:26.431972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.431970    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:17:26.432146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.431980    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:17:26.432146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.431990    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h825p\" (UniqueName: \"kubernetes.io/projected/fc97e533-efaa-416e-890c-d16bc9311ce5-kube-api-access-h825p\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:17:26.432146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.431999    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc97e533-efaa-416e-890c-d16bc9311ce5-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:17:26.609200 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.609090    2551 generic.go:356] "Generic (PLEG): container finished" podID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerID="e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a" exitCode=0
Jul 29 11:17:26.609200 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.609155    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerDied","Data":"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"}
Jul 29 11:17:26.609200 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.609177    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"
Jul 29 11:17:26.609422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.609226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs" event={"ID":"fc97e533-efaa-416e-890c-d16bc9311ce5","Type":"ContainerDied","Data":"8fdcec6ae77cbeab5ef1f3ecf8592b762644cd2686d4670258e6fc7440d44bf1"}
Jul 29 11:17:26.609422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.609254    2551 scope.go:117] "RemoveContainer" containerID="5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"
Jul 29 11:17:26.617522 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.617499    2551 scope.go:117] "RemoveContainer" containerID="e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"
Jul 29 11:17:26.625000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.624982    2551 scope.go:117] "RemoveContainer" containerID="2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"
Jul 29 11:17:26.631708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.631686    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:17:26.632322 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.632306    2551 scope.go:117] "RemoveContainer" containerID="5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"
Jul 29 11:17:26.632580 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:17:26.632559    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db\": container with ID starting with 5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db not found: ID does not exist" containerID="5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"
Jul 29 11:17:26.632630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.632589    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db"} err="failed to get container status \"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db\": rpc error: code = NotFound desc = could not find container \"5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db\": container with ID starting with 5d885999ddb8a57950eccf45e82e6d39188f7aadf066ec2facab788cea9bb8db not found: ID does not exist"
Jul 29 11:17:26.632630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.632607    2551 scope.go:117] "RemoveContainer" containerID="e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"
Jul 29 11:17:26.632863 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:17:26.632843    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a\": container with ID starting with e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a not found: ID does not exist" containerID="e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"
Jul 29 11:17:26.632917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.632868    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a"} err="failed to get container status \"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a\": rpc error: code = NotFound desc = could not find container \"e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a\": container with ID starting with e02f9ca02f437ef89cffdb8892c4ba6934d4925b763a29c123c84fdf7e654d1a not found: ID does not exist"
Jul 29 11:17:26.632917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.632884    2551 scope.go:117] "RemoveContainer" containerID="2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"
Jul 29 11:17:26.633103 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:17:26.633090    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc\": container with ID starting with 2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc not found: ID does not exist" containerID="2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"
Jul 29 11:17:26.633175 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.633106    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc"} err="failed to get container status \"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc\": rpc error: code = NotFound desc = could not find container \"2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc\": container with ID starting with 2489b03e996457a22f868cf13050eac50e1939da4f2e2e10e79ae602e5dd38dc not found: ID does not exist"
Jul 29 11:17:26.637868 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.637848    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-p9wcs"]
Jul 29 11:17:26.728285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:26.728254    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" path="/var/lib/kubelet/pods/fc97e533-efaa-416e-890c-d16bc9311ce5/volumes"
Jul 29 11:17:29.025301 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025265    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025593    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="storage-initializer"
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025604    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="storage-initializer"
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025628    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kube-rbac-proxy"
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025633    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kube-rbac-proxy"
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025645    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container"
Jul 29 11:17:29.025677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025651    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container"
Jul 29 11:17:29.025872 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025700    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kserve-container"
Jul 29 11:17:29.025872 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.025707    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc97e533-efaa-416e-890c-d16bc9311ce5" containerName="kube-rbac-proxy"
Jul 29 11:17:29.030314 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.030296    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.032765 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.032742    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 29 11:17:29.033043 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033019    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:17:29.033185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033063    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 29 11:17:29.033185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033072    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:17:29.033185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033163    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:17:29.033185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033177    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:17:29.033415 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.033262    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:17:29.040061 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.040041    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:17:29.049266 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.049244    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cds\" (UniqueName: \"kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.049370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.049291    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.049370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.049332    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.049370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.049349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.049477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.049415    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149770 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.149743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.149780    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.149797    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.149839    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.149859    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cds\" (UniqueName: \"kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.149924 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:17:29.149908    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 29 11:17:29.150167 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:17:29.149972    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls podName:981b5ebb-70cb-4764-b157-227e086500a5 nodeName:}" failed. No retries permitted until 2026-07-29 11:17:29.649950128 +0000 UTC m=+2901.552178520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" (UID: "981b5ebb-70cb-4764-b157-227e086500a5") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 29 11:17:29.150257 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.150236    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.150640 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.150619    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.150677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.150640    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.160497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.160474    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cds\" (UniqueName: \"kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.652682 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.652651    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.655101 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.655069    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:29.940840 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:29.940771    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:30.069455 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:30.069430    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:17:30.071465 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:17:30.071435    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981b5ebb_70cb_4764_b157_227e086500a5.slice/crio-d0c394cac23d0a75eb99adef1546569548ed07d8413f5451e555d9ad9b53c43b WatchSource:0}: Error finding container d0c394cac23d0a75eb99adef1546569548ed07d8413f5451e555d9ad9b53c43b: Status 404 returned error can't find the container with id d0c394cac23d0a75eb99adef1546569548ed07d8413f5451e555d9ad9b53c43b
Jul 29 11:17:30.629044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:30.629006    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerStarted","Data":"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"}
Jul 29 11:17:30.629254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:30.629075    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerStarted","Data":"d0c394cac23d0a75eb99adef1546569548ed07d8413f5451e555d9ad9b53c43b"}
Jul 29 11:17:34.647286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:34.647204    2551 generic.go:356] "Generic (PLEG): container finished" podID="981b5ebb-70cb-4764-b157-227e086500a5" containerID="4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a" exitCode=0
Jul 29 11:17:34.647286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:34.647241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerDied","Data":"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"}
Jul 29 11:17:35.653777 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:35.653742    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerStarted","Data":"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"}
Jul 29 11:17:35.654171 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:35.653808    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerStarted","Data":"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"}
Jul 29 11:17:35.654171 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:35.653989    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:35.674123 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:35.674083    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podStartSLOduration=2.427082206 podStartE2EDuration="6.674069126s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.24698692s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:17:35.672752388 +0000 UTC m=+2907.574980798" watchObservedRunningTime="2026-07-29 11:17:35.674069126 +0000 UTC m=+2907.576297574"
Jul 29 11:17:36.658866 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:36.658836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:36.660080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:36.660055    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:17:37.662815 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:37.662773    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:17:42.666935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:42.666902    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:17:42.667519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:42.667494    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:17:52.667767 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:17:52.667723    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:18:02.667695 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:02.667653    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:18:12.667526 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:12.667484    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:18:22.668280 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:22.668250    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:18:30.949836 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:30.949641    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:18:30.950278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:30.950203    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" containerID="cri-o://586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948" gracePeriod=30
Jul 29 11:18:30.950552 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:30.950503    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kube-rbac-proxy" containerID="cri-o://037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7" gracePeriod=30
Jul 29 11:18:31.888853 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:31.888820    2551 generic.go:356] "Generic (PLEG): container finished" podID="981b5ebb-70cb-4764-b157-227e086500a5" containerID="037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7" exitCode=2
Jul 29 11:18:31.889031 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:31.888898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerDied","Data":"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"}
Jul 29 11:18:32.663549 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:32.663503    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 29 11:18:32.667863 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:32.667838    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 29 11:18:33.299923 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.299900    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:18:33.386377 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386316    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") pod \"981b5ebb-70cb-4764-b157-227e086500a5\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") "
Jul 29 11:18:33.386377 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386356    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert\") pod \"981b5ebb-70cb-4764-b157-227e086500a5\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") "
Jul 29 11:18:33.386571 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386395    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9cds\" (UniqueName: \"kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds\") pod \"981b5ebb-70cb-4764-b157-227e086500a5\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") "
Jul 29 11:18:33.386571 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386416    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config\") pod \"981b5ebb-70cb-4764-b157-227e086500a5\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") "
Jul 29 11:18:33.386571 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386466    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location\") pod \"981b5ebb-70cb-4764-b157-227e086500a5\" (UID: \"981b5ebb-70cb-4764-b157-227e086500a5\") "
Jul 29 11:18:33.386832 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386806    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "981b5ebb-70cb-4764-b157-227e086500a5" (UID: "981b5ebb-70cb-4764-b157-227e086500a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:18:33.386914 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.386855    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "981b5ebb-70cb-4764-b157-227e086500a5" (UID: "981b5ebb-70cb-4764-b157-227e086500a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:18:33.388433 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.388408    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds" (OuterVolumeSpecName: "kube-api-access-q9cds") pod "981b5ebb-70cb-4764-b157-227e086500a5" (UID: "981b5ebb-70cb-4764-b157-227e086500a5"). InnerVolumeSpecName "kube-api-access-q9cds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:18:33.388825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.388803    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "981b5ebb-70cb-4764-b157-227e086500a5" (UID: "981b5ebb-70cb-4764-b157-227e086500a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:18:33.397307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.397284    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "981b5ebb-70cb-4764-b157-227e086500a5" (UID: "981b5ebb-70cb-4764-b157-227e086500a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:18:33.487024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.486998    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/981b5ebb-70cb-4764-b157-227e086500a5-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:18:33.487024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.487021    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/981b5ebb-70cb-4764-b157-227e086500a5-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:18:33.487190 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.487037    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:18:33.487190 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.487049    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9cds\" (UniqueName: \"kubernetes.io/projected/981b5ebb-70cb-4764-b157-227e086500a5-kube-api-access-q9cds\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:18:33.487190 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.487061    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/981b5ebb-70cb-4764-b157-227e086500a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:18:33.899381 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.899354    2551 generic.go:356] "Generic (PLEG): container finished" podID="981b5ebb-70cb-4764-b157-227e086500a5" containerID="586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948" exitCode=0
Jul 29 11:18:33.899778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.899386    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerDied","Data":"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"}
Jul 29 11:18:33.899778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.899414    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"
Jul 29 11:18:33.899778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.899429    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc" event={"ID":"981b5ebb-70cb-4764-b157-227e086500a5","Type":"ContainerDied","Data":"d0c394cac23d0a75eb99adef1546569548ed07d8413f5451e555d9ad9b53c43b"}
Jul 29 11:18:33.899778 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.899454    2551 scope.go:117] "RemoveContainer" containerID="037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"
Jul 29 11:18:33.907965 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.907935    2551 scope.go:117] "RemoveContainer" containerID="586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"
Jul 29 11:18:33.915092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.915073    2551 scope.go:117] "RemoveContainer" containerID="4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"
Jul 29 11:18:33.921513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.921491    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:18:33.924191 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.924174    2551 scope.go:117] "RemoveContainer" containerID="037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"
Jul 29 11:18:33.925098 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:18:33.925070    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7\": container with ID starting with 037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7 not found: ID does not exist" containerID="037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"
Jul 29 11:18:33.925222 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.925109    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7"} err="failed to get container status \"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7\": rpc error: code = NotFound desc = could not find container \"037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7\": container with ID starting with 037c22e3b9e23764c44291ad2ab5052b92d9aa068886a21a7b4aef4667a44ce7 not found: ID does not exist"
Jul 29 11:18:33.925222 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.925150    2551 scope.go:117] "RemoveContainer" containerID="586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"
Jul 29 11:18:33.925584 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:18:33.925563    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948\": container with ID starting with 586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948 not found: ID does not exist" containerID="586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"
Jul 29 11:18:33.925649 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.925593    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948"} err="failed to get container status \"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948\": rpc error: code = NotFound desc = could not find container \"586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948\": container with ID starting with 586e65857e664c833c7d7fba42ed044161e66d814812d391c559fdfe1171a948 not found: ID does not exist"
Jul 29 11:18:33.925649 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.925615    2551 scope.go:117] "RemoveContainer" containerID="4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"
Jul 29 11:18:33.925897 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:18:33.925876    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a\": container with ID starting with 4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a not found: ID does not exist" containerID="4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"
Jul 29 11:18:33.925939 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.925909    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a"} err="failed to get container status \"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a\": rpc error: code = NotFound desc = could not find container \"4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a\": container with ID starting with 4f89a9ac62825ca8b6c3a42a89c7e4d645391f0154e1c146e124cfcfd063fd6a not found: ID does not exist"
Jul 29 11:18:33.926838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:33.926821    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-nfzqc"]
Jul 29 11:18:34.728740 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:34.728710    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981b5ebb-70cb-4764-b157-227e086500a5" path="/var/lib/kubelet/pods/981b5ebb-70cb-4764-b157-227e086500a5/volumes"
Jul 29 11:18:36.129755 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.129725    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:18:36.130271 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130253    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="storage-initializer"
Jul 29 11:18:36.130361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130274    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="storage-initializer"
Jul 29 11:18:36.130361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130287    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container"
Jul 29 11:18:36.130361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130295    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container"
Jul 29 11:18:36.130361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130306    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kube-rbac-proxy"
Jul 29 11:18:36.130361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130315    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kube-rbac-proxy"
Jul 29 11:18:36.130615 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130395    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kube-rbac-proxy"
Jul 29 11:18:36.130615 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.130410    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="981b5ebb-70cb-4764-b157-227e086500a5" containerName="kserve-container"
Jul 29 11:18:36.134767 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.134747    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.137279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.137259    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 29 11:18:36.137381 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.137343    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 29 11:18:36.137381 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.137365    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:18:36.137499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.137399    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:18:36.137499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.137365    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:18:36.138346 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.138323    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:18:36.138460 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.138328    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:18:36.147708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.147687    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:18:36.206701 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.206682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.206823 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.206718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.206823 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.206807    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.206920 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.206873    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.206920 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.206900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wwhv\" (UniqueName: \"kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.307547 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.307527    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.307645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.307560    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.307645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.307577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wwhv\" (UniqueName: \"kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.307645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.307604    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.307645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.307623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.308097 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.308072    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.308264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.308246    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.308337 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.308321    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.309982 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.309961    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.315811 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.315777    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wwhv\" (UniqueName: \"kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv\") pod \"isvc-pmml-predictor-8674795f59-tvh57\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.445182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.445100    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:36.574769 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.574735    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:18:36.577721 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:18:36.577695    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912d226d_66ae_4bb1_a600_7977ab1db903.slice/crio-bf8a74469326b5e085bc42c8543423fec90562eaa4502e44d36545b45c9d6ac5 WatchSource:0}: Error finding container bf8a74469326b5e085bc42c8543423fec90562eaa4502e44d36545b45c9d6ac5: Status 404 returned error can't find the container with id bf8a74469326b5e085bc42c8543423fec90562eaa4502e44d36545b45c9d6ac5
Jul 29 11:18:36.913364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.913325    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerStarted","Data":"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"}
Jul 29 11:18:36.914373 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:36.913762    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerStarted","Data":"bf8a74469326b5e085bc42c8543423fec90562eaa4502e44d36545b45c9d6ac5"}
Jul 29 11:18:40.929255 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:40.929224    2551 generic.go:356] "Generic (PLEG): container finished" podID="912d226d-66ae-4bb1-a600-7977ab1db903" containerID="541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a" exitCode=0
Jul 29 11:18:40.929694 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:40.929305    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerDied","Data":"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"}
Jul 29 11:18:47.972764 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:47.972719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerStarted","Data":"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"}
Jul 29 11:18:48.979339 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:48.979298    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerStarted","Data":"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"}
Jul 29 11:18:48.979813 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:48.979409    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:49.015880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:49.015832    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podStartSLOduration=2.255549655 podStartE2EDuration="13.015817235s" totalImagesPullingTime="6.972116013s" totalInitContainerRuntime="3.788151567s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:18:49.013934267 +0000 UTC m=+2980.916162664" watchObservedRunningTime="2026-07-29 11:18:49.015817235 +0000 UTC m=+2980.918045631"
Jul 29 11:18:49.987225 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:49.987185    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:49.990850 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:49.990812    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:18:50.988939 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:50.988898    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:18:55.993277 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:55.993244    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:18:55.993912 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:18:55.993883    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:05.994045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:05.994005    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:08.812669 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:08.812643    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:19:08.820009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:08.819988    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:19:15.994215 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:15.994177    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:25.993921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:25.993885    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:35.993797 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:35.993757    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:45.994403 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:45.994356    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:19:55.994521 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:19:55.994479    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 29 11:20:05.995149 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:05.995028    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:20:17.210523 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:17.210493    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:20:17.210996 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:17.210730    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container" containerID="cri-o://b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a" gracePeriod=30
Jul 29 11:20:17.210996 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:17.210799    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kube-rbac-proxy" containerID="cri-o://d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2" gracePeriod=30
Jul 29 11:20:17.359269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:17.359240    2551 generic.go:356] "Generic (PLEG): container finished" podID="912d226d-66ae-4bb1-a600-7977ab1db903" containerID="d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2" exitCode=2
Jul 29 11:20:17.359378 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:17.359299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerDied","Data":"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"}
Jul 29 11:20:20.554869 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.554838    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:20:20.706166 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706123    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert\") pod \"912d226d-66ae-4bb1-a600-7977ab1db903\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") "
Jul 29 11:20:20.706317 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706182    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config\") pod \"912d226d-66ae-4bb1-a600-7977ab1db903\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") "
Jul 29 11:20:20.706317 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706201    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls\") pod \"912d226d-66ae-4bb1-a600-7977ab1db903\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") "
Jul 29 11:20:20.706317 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706235    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location\") pod \"912d226d-66ae-4bb1-a600-7977ab1db903\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") "
Jul 29 11:20:20.706317 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706256    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wwhv\" (UniqueName: \"kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv\") pod \"912d226d-66ae-4bb1-a600-7977ab1db903\" (UID: \"912d226d-66ae-4bb1-a600-7977ab1db903\") "
Jul 29 11:20:20.706526 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706435    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "912d226d-66ae-4bb1-a600-7977ab1db903" (UID: "912d226d-66ae-4bb1-a600-7977ab1db903"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:20:20.706601 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706580    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "912d226d-66ae-4bb1-a600-7977ab1db903" (UID: "912d226d-66ae-4bb1-a600-7977ab1db903"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:20:20.706664 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.706593    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "912d226d-66ae-4bb1-a600-7977ab1db903" (UID: "912d226d-66ae-4bb1-a600-7977ab1db903"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:20:20.708243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.708221    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "912d226d-66ae-4bb1-a600-7977ab1db903" (UID: "912d226d-66ae-4bb1-a600-7977ab1db903"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:20:20.708419 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.708402    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv" (OuterVolumeSpecName: "kube-api-access-6wwhv") pod "912d226d-66ae-4bb1-a600-7977ab1db903" (UID: "912d226d-66ae-4bb1-a600-7977ab1db903"). InnerVolumeSpecName "kube-api-access-6wwhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:20:20.807273 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.807237    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/912d226d-66ae-4bb1-a600-7977ab1db903-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:20:20.807273 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.807270    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wwhv\" (UniqueName: \"kubernetes.io/projected/912d226d-66ae-4bb1-a600-7977ab1db903-kube-api-access-6wwhv\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:20:20.807273 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.807280    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:20:20.807446 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.807290    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/912d226d-66ae-4bb1-a600-7977ab1db903-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:20:20.807446 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:20.807299    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/912d226d-66ae-4bb1-a600-7977ab1db903-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:20:21.377643 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.377605    2551 generic.go:356] "Generic (PLEG): container finished" podID="912d226d-66ae-4bb1-a600-7977ab1db903" containerID="b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a" exitCode=0
Jul 29 11:20:21.377816 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.377659    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"
Jul 29 11:20:21.377816 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.377659    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerDied","Data":"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"}
Jul 29 11:20:21.377816 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.377778    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57" event={"ID":"912d226d-66ae-4bb1-a600-7977ab1db903","Type":"ContainerDied","Data":"bf8a74469326b5e085bc42c8543423fec90562eaa4502e44d36545b45c9d6ac5"}
Jul 29 11:20:21.377816 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.377806    2551 scope.go:117] "RemoveContainer" containerID="d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"
Jul 29 11:20:21.385969 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.385945    2551 scope.go:117] "RemoveContainer" containerID="b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"
Jul 29 11:20:21.393240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.393208    2551 scope.go:117] "RemoveContainer" containerID="541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"
Jul 29 11:20:21.394712 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.394693    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:20:21.399570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.399550    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tvh57"]
Jul 29 11:20:21.400475 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.400461    2551 scope.go:117] "RemoveContainer" containerID="d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"
Jul 29 11:20:21.400739 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:20:21.400724    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2\": container with ID starting with d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2 not found: ID does not exist" containerID="d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"
Jul 29 11:20:21.400785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.400747    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2"} err="failed to get container status \"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2\": rpc error: code = NotFound desc = could not find container \"d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2\": container with ID starting with d1c1999c8f3bfee65feb9e54fb1e9b02f60820850cbecd0e07863edf9c29a8c2 not found: ID does not exist"
Jul 29 11:20:21.400785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.400764    2551 scope.go:117] "RemoveContainer" containerID="b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"
Jul 29 11:20:21.400996 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:20:21.400971    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a\": container with ID starting with b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a not found: ID does not exist" containerID="b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"
Jul 29 11:20:21.401041 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.401003    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a"} err="failed to get container status \"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a\": rpc error: code = NotFound desc = could not find container \"b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a\": container with ID starting with b887c52a2e1671be6fa7316e24959ed730331e805c8466d2273f7c91de73645a not found: ID does not exist"
Jul 29 11:20:21.401041 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.401019    2551 scope.go:117] "RemoveContainer" containerID="541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"
Jul 29 11:20:21.401387 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:20:21.401369    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a\": container with ID starting with 541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a not found: ID does not exist" containerID="541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"
Jul 29 11:20:21.401452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:21.401390    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a"} err="failed to get container status \"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a\": rpc error: code = NotFound desc = could not find container \"541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a\": container with ID starting with 541388133febbe2780cdc51df08719970488090c5fb9eaf288d2f7eae944804a not found: ID does not exist"
Jul 29 11:20:22.357189 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357153    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357467    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kube-rbac-proxy"
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357477    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kube-rbac-proxy"
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357490    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="storage-initializer"
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357495    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="storage-initializer"
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357513    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container"
Jul 29 11:20:22.357561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357518    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container"
Jul 29 11:20:22.357756 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357568    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kube-rbac-proxy"
Jul 29 11:20:22.357756 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.357577    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" containerName="kserve-container"
Jul 29 11:20:22.362014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.361997    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.364544 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.364518    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:20:22.364681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.364563    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 29 11:20:22.364681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.364625    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:20:22.364820 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.364678    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:20:22.364820 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.364794    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:20:22.365525 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.365506    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:20:22.365644 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.365542    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:20:22.371498 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.371476    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:20:22.522544 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.522514    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.522727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.522564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.522727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.522614    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xqp\" (UniqueName: \"kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.522727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.522714    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.522889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.522769    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.623864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.623784    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.623864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.623827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.623864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.623852    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.624113 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.623869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xqp\" (UniqueName: \"kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.624113 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.623919    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.624399 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.624371    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.624532 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.624482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.624607 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.624578    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.626312 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.626292    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.631948 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.631929    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xqp\" (UniqueName: \"kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp\") pod \"isvc-pmml-runtime-predictor-885fc8777-htlg7\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.672914 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.672889    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:22.730168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.729751    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="912d226d-66ae-4bb1-a600-7977ab1db903" path="/var/lib/kubelet/pods/912d226d-66ae-4bb1-a600-7977ab1db903/volumes"
Jul 29 11:20:22.804646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:22.804612    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:20:22.809337 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:20:22.809306    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ee0a61_37f1_49da_aca0_21810ad0b977.slice/crio-3e5fec8587c29b8bb6fbd67be01489949d46a7a718d7848e1aa96eec518b6246 WatchSource:0}: Error finding container 3e5fec8587c29b8bb6fbd67be01489949d46a7a718d7848e1aa96eec518b6246: Status 404 returned error can't find the container with id 3e5fec8587c29b8bb6fbd67be01489949d46a7a718d7848e1aa96eec518b6246
Jul 29 11:20:23.389674 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:23.389589    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerStarted","Data":"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"}
Jul 29 11:20:23.389674 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:23.389647    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerStarted","Data":"3e5fec8587c29b8bb6fbd67be01489949d46a7a718d7848e1aa96eec518b6246"}
Jul 29 11:20:27.409736 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:27.409702    2551 generic.go:356] "Generic (PLEG): container finished" podID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerID="97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1" exitCode=0
Jul 29 11:20:27.410226 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:27.409776    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerDied","Data":"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"}
Jul 29 11:20:28.416590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.416555    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerStarted","Data":"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"}
Jul 29 11:20:28.416974 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.416617    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerStarted","Data":"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"}
Jul 29 11:20:28.416974 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.416884    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:28.417048 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.417009    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:28.418326 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.418301    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:20:28.449338 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:28.449293    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podStartSLOduration=2.5878794320000003 podStartE2EDuration="6.449279762s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86140033s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:20:28.447834966 +0000 UTC m=+3080.350063364" watchObservedRunningTime="2026-07-29 11:20:28.449279762 +0000 UTC m=+3080.351508159"
Jul 29 11:20:29.426293 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:29.426251    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:20:34.430688 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:34.430660    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:20:34.431149 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:34.431108    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:20:44.431980 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:44.431937    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:20:54.431218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:20:54.431179    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:04.431214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:04.431177    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:14.431551 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:14.431514    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:24.431214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:24.431176    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:34.431374 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:34.431331    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:44.431760 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:44.431731    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:21:53.355639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:53.355607    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:21:53.356063 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:53.355839    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" containerID="cri-o://c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d" gracePeriod=30
Jul 29 11:21:53.356063 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:53.355872    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kube-rbac-proxy" containerID="cri-o://475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb" gracePeriod=30
Jul 29 11:21:53.794301 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:53.794268    2551 generic.go:356] "Generic (PLEG): container finished" podID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerID="475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb" exitCode=2
Jul 29 11:21:53.794471 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:53.794307    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerDied","Data":"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"}
Jul 29 11:21:54.426995 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:54.426952    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 29 11:21:54.431434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:54.431406    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 29 11:21:56.496315 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.496292    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:21:56.649456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649393    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config\") pod \"50ee0a61-37f1-49da-aca0-21810ad0b977\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") "
Jul 29 11:21:56.649456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649434    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location\") pod \"50ee0a61-37f1-49da-aca0-21810ad0b977\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") "
Jul 29 11:21:56.649456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649450    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert\") pod \"50ee0a61-37f1-49da-aca0-21810ad0b977\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") "
Jul 29 11:21:56.649700 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649516    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls\") pod \"50ee0a61-37f1-49da-aca0-21810ad0b977\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") "
Jul 29 11:21:56.649700 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649543    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xqp\" (UniqueName: \"kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp\") pod \"50ee0a61-37f1-49da-aca0-21810ad0b977\" (UID: \"50ee0a61-37f1-49da-aca0-21810ad0b977\") "
Jul 29 11:21:56.649808 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649751    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50ee0a61-37f1-49da-aca0-21810ad0b977" (UID: "50ee0a61-37f1-49da-aca0-21810ad0b977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:21:56.649808 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649784    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50ee0a61-37f1-49da-aca0-21810ad0b977" (UID: "50ee0a61-37f1-49da-aca0-21810ad0b977"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:21:56.649930 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.649843    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50ee0a61-37f1-49da-aca0-21810ad0b977" (UID: "50ee0a61-37f1-49da-aca0-21810ad0b977"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:21:56.651685 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.651663    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50ee0a61-37f1-49da-aca0-21810ad0b977" (UID: "50ee0a61-37f1-49da-aca0-21810ad0b977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:21:56.651759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.651698    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp" (OuterVolumeSpecName: "kube-api-access-c4xqp") pod "50ee0a61-37f1-49da-aca0-21810ad0b977" (UID: "50ee0a61-37f1-49da-aca0-21810ad0b977"). InnerVolumeSpecName "kube-api-access-c4xqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:21:56.750233 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.750204    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50ee0a61-37f1-49da-aca0-21810ad0b977-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:21:56.750233 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.750232    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xqp\" (UniqueName: \"kubernetes.io/projected/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-api-access-c4xqp\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:21:56.750233 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.750242    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:21:56.750389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.750252    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50ee0a61-37f1-49da-aca0-21810ad0b977-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:21:56.750389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.750260    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50ee0a61-37f1-49da-aca0-21810ad0b977-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:21:56.807136 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.807104    2551 generic.go:356] "Generic (PLEG): container finished" podID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerID="c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d" exitCode=0
Jul 29 11:21:56.807244 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.807150    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerDied","Data":"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"}
Jul 29 11:21:56.807244 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.807184    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"
Jul 29 11:21:56.807244 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.807197    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7" event={"ID":"50ee0a61-37f1-49da-aca0-21810ad0b977","Type":"ContainerDied","Data":"3e5fec8587c29b8bb6fbd67be01489949d46a7a718d7848e1aa96eec518b6246"}
Jul 29 11:21:56.807244 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.807222    2551 scope.go:117] "RemoveContainer" containerID="475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"
Jul 29 11:21:56.815167 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.815149    2551 scope.go:117] "RemoveContainer" containerID="c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"
Jul 29 11:21:56.822856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.822664    2551 scope.go:117] "RemoveContainer" containerID="97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"
Jul 29 11:21:56.824555 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.824538    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:21:56.830257 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830238    2551 scope.go:117] "RemoveContainer" containerID="475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"
Jul 29 11:21:56.830468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830451    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-htlg7"]
Jul 29 11:21:56.830523 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:21:56.830488    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb\": container with ID starting with 475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb not found: ID does not exist" containerID="475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"
Jul 29 11:21:56.830523 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830507    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb"} err="failed to get container status \"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb\": rpc error: code = NotFound desc = could not find container \"475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb\": container with ID starting with 475363be0c60ea961e5e55fad543b1470d0bd22a0da580570476df605ed39afb not found: ID does not exist"
Jul 29 11:21:56.830523 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830521    2551 scope.go:117] "RemoveContainer" containerID="c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"
Jul 29 11:21:56.830713 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:21:56.830697    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d\": container with ID starting with c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d not found: ID does not exist" containerID="c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"
Jul 29 11:21:56.830754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830719    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d"} err="failed to get container status \"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d\": rpc error: code = NotFound desc = could not find container \"c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d\": container with ID starting with c2d71481276f3b99b9938a0c7e2496f44ddf867379d177a842170094da83740d not found: ID does not exist"
Jul 29 11:21:56.830754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830735    2551 scope.go:117] "RemoveContainer" containerID="97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"
Jul 29 11:21:56.830957 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:21:56.830939    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1\": container with ID starting with 97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1 not found: ID does not exist" containerID="97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"
Jul 29 11:21:56.831010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:56.830962    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1"} err="failed to get container status \"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1\": rpc error: code = NotFound desc = could not find container \"97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1\": container with ID starting with 97c7e645bb81cc612f166dd87512a5038f25a6c2f585adaaeebe978a73411dd1 not found: ID does not exist"
Jul 29 11:21:58.728291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:21:58.728250    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" path="/var/lib/kubelet/pods/50ee0a61-37f1-49da-aca0-21810ad0b977/volumes"
Jul 29 11:23:44.734040 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734007    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734334    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734345    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734362    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="storage-initializer"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734368    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="storage-initializer"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734375    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kube-rbac-proxy"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734380    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kube-rbac-proxy"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734441    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kube-rbac-proxy"
Jul 29 11:23:44.734509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.734449    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ee0a61-37f1-49da-aca0-21810ad0b977" containerName="kserve-container"
Jul 29 11:23:44.737514 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.737499    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:44.740009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.739981    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2d1343-predictor-serving-cert\""
Jul 29 11:23:44.740009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.740000    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:23:44.740202 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.740013    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2d1343-kube-rbac-proxy-sar-config\""
Jul 29 11:23:44.740202 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.740080    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:23:44.741030 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.741014    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:23:44.741086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.741053    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:23:44.741376 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.741333    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:23:44.753371 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.750725    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:23:44.908408 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.908381    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fzr\" (UniqueName: \"kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:44.908512 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.908416    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:44.908512 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.908436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:44.908594 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.908570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:44.908631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:44.908603    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009370    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fzr\" (UniqueName: \"kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009402    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009420    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009470    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009500    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.009834 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.009815    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.010088 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.010071    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.010221 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.010202    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.012118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.012091    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.017661 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.017638    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fzr\" (UniqueName: \"kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr\") pod \"isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") " pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.047163 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.047120    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:45.176763 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.176724    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:23:45.179571 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:23:45.179547    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aedc461_8038_4ddd_ac79_cd25e7088592.slice/crio-17ac3c1c4144a53ef5c6c19b87adade848d782a5204d06f1bdaeac0b9176f807 WatchSource:0}: Error finding container 17ac3c1c4144a53ef5c6c19b87adade848d782a5204d06f1bdaeac0b9176f807: Status 404 returned error can't find the container with id 17ac3c1c4144a53ef5c6c19b87adade848d782a5204d06f1bdaeac0b9176f807
Jul 29 11:23:45.282052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.281985    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerStarted","Data":"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"}
Jul 29 11:23:45.282052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:45.282039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerStarted","Data":"17ac3c1c4144a53ef5c6c19b87adade848d782a5204d06f1bdaeac0b9176f807"}
Jul 29 11:23:49.299431 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:49.299402    2551 generic.go:356] "Generic (PLEG): container finished" podID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerID="08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21" exitCode=0
Jul 29 11:23:49.299792 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:49.299439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerDied","Data":"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"}
Jul 29 11:23:50.312281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.312240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerStarted","Data":"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"}
Jul 29 11:23:50.312680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.312324    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerStarted","Data":"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"}
Jul 29 11:23:50.312680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.312632    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:50.312760 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.312698    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:50.313964 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.313938    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:23:50.333455 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:50.333412    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podStartSLOduration=2.4340921 podStartE2EDuration="6.33339944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89930734s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:23:50.331816496 +0000 UTC m=+3282.234044884" watchObservedRunningTime="2026-07-29 11:23:50.33339944 +0000 UTC m=+3282.235627837"
Jul 29 11:23:51.317480 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:51.317435    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:23:56.321925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:56.321894    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:23:56.322558 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:23:56.322524    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:06.323434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:06.323392    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:08.840454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:08.840425    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:24:08.848906 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:08.848886    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:24:16.322509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:16.322467    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:26.322568 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:26.322529    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:36.322907 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:36.322818    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:46.322817 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:46.322771    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:24:56.322963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:24:56.322921    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 29 11:25:02.728421 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:02.728386    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:25:19.932942 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.932909    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:19.936278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.936262    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:19.938845 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.938821    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2d1343-predictor-serving-cert\""
Jul 29 11:25:19.938845 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.938824    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-2d1343-dockercfg-c44m5\""
Jul 29 11:25:19.939035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.938830    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-2d1343\""
Jul 29 11:25:19.939035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.938894    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2d1343-kube-rbac-proxy-sar-config\""
Jul 29 11:25:19.946863 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:19.946845    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:20.094060 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.094037    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.094171 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.094086    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.094218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.094183    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.094218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.094213    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspk5\" (UniqueName: \"kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.094291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.094231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195091 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195025    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195091 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195062    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspk5\" (UniqueName: \"kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195091 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195081    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195152    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195209    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195636    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195832 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195812    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.195878 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.195837    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.197808 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.197786    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.202539 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.202516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspk5\" (UniqueName: \"kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5\") pod \"isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") " pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.247474 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.247451    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:20.374402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.374378    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:20.376473 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:25:20.376439    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbd4187_3038_4e1f_9e6c_c241acb0053d.slice/crio-0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de WatchSource:0}: Error finding container 0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de: Status 404 returned error can't find the container with id 0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de
Jul 29 11:25:20.693693 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.693658    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerStarted","Data":"e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"}
Jul 29 11:25:20.693887 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:20.693724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerStarted","Data":"0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de"}
Jul 29 11:25:26.723865 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:26.723824    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/0.log"
Jul 29 11:25:26.724310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:26.723870    2551 generic.go:356] "Generic (PLEG): container finished" podID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerID="e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987" exitCode=1
Jul 29 11:25:26.728360 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:26.728322    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerDied","Data":"e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"}
Jul 29 11:25:27.730974 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:27.730943    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/0.log"
Jul 29 11:25:27.731376 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:27.731071    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerStarted","Data":"9977448544b34e1a113b7530bae87c250ab30af8d2fe89f61b71b6b702bce779"}
Jul 29 11:25:30.747633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.747603    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/1.log"
Jul 29 11:25:30.748066 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.747969    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/0.log"
Jul 29 11:25:30.748066 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.748002    2551 generic.go:356] "Generic (PLEG): container finished" podID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerID="9977448544b34e1a113b7530bae87c250ab30af8d2fe89f61b71b6b702bce779" exitCode=1
Jul 29 11:25:30.748173 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.748073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerDied","Data":"9977448544b34e1a113b7530bae87c250ab30af8d2fe89f61b71b6b702bce779"}
Jul 29 11:25:30.748173 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.748163    2551 scope.go:117] "RemoveContainer" containerID="e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"
Jul 29 11:25:30.748561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.748540    2551 scope.go:117] "RemoveContainer" containerID="e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"
Jul 29 11:25:30.759048 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:30.759013    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_kserve-ci-e2e-test(3cbd4187-3038-4e1f-9e6c-c241acb0053d)\"" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d"
Jul 29 11:25:30.759330 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:30.759304    2551 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_kserve-ci-e2e-test_3cbd4187-3038-4e1f-9e6c-c241acb0053d_0 in pod sandbox 0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de from index: no such id: 'e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987'" containerID="e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"
Jul 29 11:25:30.759394 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:30.759342    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_kserve-ci-e2e-test_3cbd4187-3038-4e1f-9e6c-c241acb0053d_0 in pod sandbox 0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de from index: no such id: 'e992d93559c70fc96d8c37c05b466f601ec18f26504949bbe1fc605ec85f5987'"
Jul 29 11:25:31.754839 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:31.754803    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/1.log"
Jul 29 11:25:37.954884 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:37.954849    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:25:37.955374 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:37.955104    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container" containerID="cri-o://a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf" gracePeriod=30
Jul 29 11:25:37.955374 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:37.955306    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kube-rbac-proxy" containerID="cri-o://bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56" gracePeriod=30
Jul 29 11:25:38.011702 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.011676    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:38.133081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.133058    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/1.log"
Jul 29 11:25:38.135387 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.135370    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:38.227655 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227603    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config\") pod \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") "
Jul 29 11:25:38.227655 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227644    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location\") pod \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") "
Jul 29 11:25:38.227819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227700    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert\") pod \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") "
Jul 29 11:25:38.227819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227729    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wspk5\" (UniqueName: \"kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5\") pod \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") "
Jul 29 11:25:38.227819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227761    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls\") pod \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\" (UID: \"3cbd4187-3038-4e1f-9e6c-c241acb0053d\") "
Jul 29 11:25:38.228012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227988    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3cbd4187-3038-4e1f-9e6c-c241acb0053d" (UID: "3cbd4187-3038-4e1f-9e6c-c241acb0053d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:25:38.228012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.227998    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3cbd4187-3038-4e1f-9e6c-c241acb0053d" (UID: "3cbd4187-3038-4e1f-9e6c-c241acb0053d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:38.228103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.228088    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3cbd4187-3038-4e1f-9e6c-c241acb0053d" (UID: "3cbd4187-3038-4e1f-9e6c-c241acb0053d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:38.229876 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.229854    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3cbd4187-3038-4e1f-9e6c-c241acb0053d" (UID: "3cbd4187-3038-4e1f-9e6c-c241acb0053d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:25:38.229930 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.229857    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5" (OuterVolumeSpecName: "kube-api-access-wspk5") pod "3cbd4187-3038-4e1f-9e6c-c241acb0053d" (UID: "3cbd4187-3038-4e1f-9e6c-c241acb0053d"). InnerVolumeSpecName "kube-api-access-wspk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:25:38.328285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.328264    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:38.328285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.328283    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:38.328408 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.328293    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3cbd4187-3038-4e1f-9e6c-c241acb0053d-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:38.328408 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.328302    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspk5\" (UniqueName: \"kubernetes.io/projected/3cbd4187-3038-4e1f-9e6c-c241acb0053d-kube-api-access-wspk5\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:38.328408 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.328310    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3cbd4187-3038-4e1f-9e6c-c241acb0053d-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:38.787497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.787466    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w_3cbd4187-3038-4e1f-9e6c-c241acb0053d/storage-initializer/1.log"
Jul 29 11:25:38.787645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.787583    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w" event={"ID":"3cbd4187-3038-4e1f-9e6c-c241acb0053d","Type":"ContainerDied","Data":"0d92508e7abee35b302112584ed4907abf820ce8d7b3b594839015137a6d67de"}
Jul 29 11:25:38.787645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.787609    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"
Jul 29 11:25:38.787780 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.787649    2551 scope.go:117] "RemoveContainer" containerID="9977448544b34e1a113b7530bae87c250ab30af8d2fe89f61b71b6b702bce779"
Jul 29 11:25:38.790330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.790305    2551 generic.go:356] "Generic (PLEG): container finished" podID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerID="bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56" exitCode=2
Jul 29 11:25:38.790443 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.790348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerDied","Data":"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"}
Jul 29 11:25:38.821410 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.821387    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:38.827710 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:38.827687    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2d1343-predictor-5d9b7b5757-lx79w"]
Jul 29 11:25:40.728270 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:40.728237    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" path="/var/lib/kubelet/pods/3cbd4187-3038-4e1f-9e6c-c241acb0053d/volumes"
Jul 29 11:25:41.318219 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.318179    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 29 11:25:41.793841 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.793819    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:25:41.804356 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.804334    2551 generic.go:356] "Generic (PLEG): container finished" podID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerID="a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf" exitCode=0
Jul 29 11:25:41.804457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.804372    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"
Jul 29 11:25:41.804457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.804395    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerDied","Data":"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"}
Jul 29 11:25:41.804457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.804444    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9" event={"ID":"2aedc461-8038-4ddd-ac79-cd25e7088592","Type":"ContainerDied","Data":"17ac3c1c4144a53ef5c6c19b87adade848d782a5204d06f1bdaeac0b9176f807"}
Jul 29 11:25:41.804560 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.804474    2551 scope.go:117] "RemoveContainer" containerID="bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"
Jul 29 11:25:41.811747 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.811731    2551 scope.go:117] "RemoveContainer" containerID="a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"
Jul 29 11:25:41.821710 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.821694    2551 scope.go:117] "RemoveContainer" containerID="08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"
Jul 29 11:25:41.829881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.829864    2551 scope.go:117] "RemoveContainer" containerID="bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"
Jul 29 11:25:41.830162 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:41.830120    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56\": container with ID starting with bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56 not found: ID does not exist" containerID="bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"
Jul 29 11:25:41.830228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.830173    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56"} err="failed to get container status \"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56\": rpc error: code = NotFound desc = could not find container \"bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56\": container with ID starting with bfbc310bfb9f47b6773a7f2eb0f0f5f4aa39f2c80bf80593e1e947d2132eff56 not found: ID does not exist"
Jul 29 11:25:41.830228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.830189    2551 scope.go:117] "RemoveContainer" containerID="a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"
Jul 29 11:25:41.830393 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:41.830375    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf\": container with ID starting with a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf not found: ID does not exist" containerID="a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"
Jul 29 11:25:41.830432 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.830397    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf"} err="failed to get container status \"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf\": rpc error: code = NotFound desc = could not find container \"a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf\": container with ID starting with a86e6e2b9d2b8770d905bba9df52b94bc80d9bea8b08c1595254f5f11806f6bf not found: ID does not exist"
Jul 29 11:25:41.830432 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.830411    2551 scope.go:117] "RemoveContainer" containerID="08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"
Jul 29 11:25:41.830613 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:41.830596    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21\": container with ID starting with 08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21 not found: ID does not exist" containerID="08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"
Jul 29 11:25:41.830694 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.830623    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21"} err="failed to get container status \"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21\": rpc error: code = NotFound desc = could not find container \"08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21\": container with ID starting with 08f9fe4eac159a4ba2e3c43c14fb78a750c3ffb99b04fb782af34cded8e2bd21 not found: ID does not exist"
Jul 29 11:25:41.952457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952429    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4fzr\" (UniqueName: \"kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr\") pod \"2aedc461-8038-4ddd-ac79-cd25e7088592\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") "
Jul 29 11:25:41.952572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952470    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert\") pod \"2aedc461-8038-4ddd-ac79-cd25e7088592\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") "
Jul 29 11:25:41.952572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952515    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config\") pod \"2aedc461-8038-4ddd-ac79-cd25e7088592\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") "
Jul 29 11:25:41.952572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls\") pod \"2aedc461-8038-4ddd-ac79-cd25e7088592\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") "
Jul 29 11:25:41.952748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952684    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location\") pod \"2aedc461-8038-4ddd-ac79-cd25e7088592\" (UID: \"2aedc461-8038-4ddd-ac79-cd25e7088592\") "
Jul 29 11:25:41.952909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2aedc461-8038-4ddd-ac79-cd25e7088592" (UID: "2aedc461-8038-4ddd-ac79-cd25e7088592"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:41.952909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952891    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2aedc461-8038-4ddd-ac79-cd25e7088592" (UID: "2aedc461-8038-4ddd-ac79-cd25e7088592"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:41.953037 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.952934    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2aedc461-8038-4ddd-ac79-cd25e7088592" (UID: "2aedc461-8038-4ddd-ac79-cd25e7088592"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:25:41.954401 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.954382    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2aedc461-8038-4ddd-ac79-cd25e7088592" (UID: "2aedc461-8038-4ddd-ac79-cd25e7088592"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:25:41.954560 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:41.954538    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr" (OuterVolumeSpecName: "kube-api-access-p4fzr") pod "2aedc461-8038-4ddd-ac79-cd25e7088592" (UID: "2aedc461-8038-4ddd-ac79-cd25e7088592"). InnerVolumeSpecName "kube-api-access-p4fzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:25:42.053638 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.053615    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aedc461-8038-4ddd-ac79-cd25e7088592-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:42.053638 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.053638    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2aedc461-8038-4ddd-ac79-cd25e7088592-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:42.053771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.053647    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fzr\" (UniqueName: \"kubernetes.io/projected/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-api-access-p4fzr\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:42.053771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.053657    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:42.053771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.053665    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aedc461-8038-4ddd-ac79-cd25e7088592-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:42.124875 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.124848    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:25:42.130933 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.130910    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2d1343-predictor-7758cf8cf8-mmjb9"]
Jul 29 11:25:42.728160 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:42.728106    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" path="/var/lib/kubelet/pods/2aedc461-8038-4ddd-ac79-cd25e7088592/volumes"
Jul 29 11:25:43.164300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164226    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164531    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="storage-initializer"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164541    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="storage-initializer"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164554    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kube-rbac-proxy"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164560    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kube-rbac-proxy"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164570    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164575    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164585    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164591    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164597    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164602    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164672    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kube-rbac-proxy"
Jul 29 11:25:43.164949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164680    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164686    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbd4187-3038-4e1f-9e6c-c241acb0053d" containerName="storage-initializer"
Jul 29 11:25:43.164949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.164693    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aedc461-8038-4ddd-ac79-cd25e7088592" containerName="kserve-container"
Jul 29 11:25:43.169600 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.169581    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.172145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.172082    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-541eb7\""
Jul 29 11:25:43.172145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.172100    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-541eb7-kube-rbac-proxy-sar-config\""
Jul 29 11:25:43.172145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.172115    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:25:43.172399 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.172183    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-541eb7-dockercfg-wr7rv\""
Jul 29 11:25:43.172399 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.172188    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-541eb7-predictor-serving-cert\""
Jul 29 11:25:43.173112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.173098    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:25:43.173112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.173106    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:25:43.177515 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.177495    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:43.362961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.362937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.363080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.362973    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbgk\" (UniqueName: \"kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.363080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.363001    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.363080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.363018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.363219 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.363076    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463661 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.463638    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.463677    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbgk\" (UniqueName: \"kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.463704    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.463721    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.463741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.463994 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:43.463852    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-serving-cert: secret "isvc-init-fail-541eb7-predictor-serving-cert" not found
Jul 29 11:25:43.463994 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:43.463932    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls podName:7c4aaf59-5591-4ec1-88e3-706b43ee9194 nodeName:}" failed. No retries permitted until 2026-07-29 11:25:43.963911032 +0000 UTC m=+3395.866139425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls") pod "isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194") : secret "isvc-init-fail-541eb7-predictor-serving-cert" not found
Jul 29 11:25:43.464161 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.464118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.464527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.464504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.464566 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.464504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.471495 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.471465    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbgk\" (UniqueName: \"kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.967831 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.967804    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:43.970118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:43.970094    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") pod \"isvc-init-fail-541eb7-predictor-66c5b888ff-ks642\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") " pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:44.080707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:44.080676    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:44.208663 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:44.208636    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:44.210786 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:25:44.210755    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4aaf59_5591_4ec1_88e3_706b43ee9194.slice/crio-b129796925f8f96d2de59a0ed1c8f43d68561bf7358827fc3050e90ff74f4fde WatchSource:0}: Error finding container b129796925f8f96d2de59a0ed1c8f43d68561bf7358827fc3050e90ff74f4fde: Status 404 returned error can't find the container with id b129796925f8f96d2de59a0ed1c8f43d68561bf7358827fc3050e90ff74f4fde
Jul 29 11:25:44.819483 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:44.819450    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerStarted","Data":"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"}
Jul 29 11:25:44.819644 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:44.819505    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerStarted","Data":"b129796925f8f96d2de59a0ed1c8f43d68561bf7358827fc3050e90ff74f4fde"}
Jul 29 11:25:48.840522 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:48.840488    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/0.log"
Jul 29 11:25:48.840918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:48.840535    2551 generic.go:356] "Generic (PLEG): container finished" podID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerID="bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5" exitCode=1
Jul 29 11:25:48.840918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:48.840623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerDied","Data":"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"}
Jul 29 11:25:49.846892 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:49.846865    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/0.log"
Jul 29 11:25:49.847268 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:49.846990    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerStarted","Data":"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"}
Jul 29 11:25:53.160946 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.160901    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:53.161447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.161172    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer" containerID="cri-o://d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b" gracePeriod=30
Jul 29 11:25:53.603344 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.602420    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/1.log"
Jul 29 11:25:53.603589 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.603562    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/0.log"
Jul 29 11:25:53.605800 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.605784    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:53.641137 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641113    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert\") pod \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") "
Jul 29 11:25:53.641256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641163    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location\") pod \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") "
Jul 29 11:25:53.641256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641183    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config\") pod \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") "
Jul 29 11:25:53.641256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641205    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") pod \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") "
Jul 29 11:25:53.641256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641229    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpbgk\" (UniqueName: \"kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk\") pod \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\" (UID: \"7c4aaf59-5591-4ec1-88e3-706b43ee9194\") "
Jul 29 11:25:53.641482 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641367    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c4aaf59-5591-4ec1-88e3-706b43ee9194" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:25:53.641545 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641516    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c4aaf59-5591-4ec1-88e3-706b43ee9194" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:53.641611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.641588    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c4aaf59-5591-4ec1-88e3-706b43ee9194" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:25:53.643276 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.643254    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c4aaf59-5591-4ec1-88e3-706b43ee9194" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:25:53.643276 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.643269    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk" (OuterVolumeSpecName: "kube-api-access-fpbgk") pod "7c4aaf59-5591-4ec1-88e3-706b43ee9194" (UID: "7c4aaf59-5591-4ec1-88e3-706b43ee9194"). InnerVolumeSpecName "kube-api-access-fpbgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:25:53.741900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.741837    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c4aaf59-5591-4ec1-88e3-706b43ee9194-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:53.741900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.741864    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbgk\" (UniqueName: \"kubernetes.io/projected/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-api-access-fpbgk\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:53.741900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.741877    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:53.741900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.741890    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:53.742077 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.741903    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c4aaf59-5591-4ec1-88e3-706b43ee9194-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:25:53.865425 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865406    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/1.log"
Jul 29 11:25:53.865829 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865814    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-541eb7-predictor-66c5b888ff-ks642_7c4aaf59-5591-4ec1-88e3-706b43ee9194/storage-initializer/0.log"
Jul 29 11:25:53.865897 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865850    2551 generic.go:356] "Generic (PLEG): container finished" podID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerID="d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b" exitCode=1
Jul 29 11:25:53.865938 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865900    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"
Jul 29 11:25:53.865973 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865931    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerDied","Data":"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"}
Jul 29 11:25:53.866021 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.865994    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642" event={"ID":"7c4aaf59-5591-4ec1-88e3-706b43ee9194","Type":"ContainerDied","Data":"b129796925f8f96d2de59a0ed1c8f43d68561bf7358827fc3050e90ff74f4fde"}
Jul 29 11:25:53.866058 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.866025    2551 scope.go:117] "RemoveContainer" containerID="d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"
Jul 29 11:25:53.874621 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.874602    2551 scope.go:117] "RemoveContainer" containerID="bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"
Jul 29 11:25:53.882101 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.882085    2551 scope.go:117] "RemoveContainer" containerID="d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"
Jul 29 11:25:53.882380 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:53.882364    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b\": container with ID starting with d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b not found: ID does not exist" containerID="d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"
Jul 29 11:25:53.882426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.882388    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b"} err="failed to get container status \"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b\": rpc error: code = NotFound desc = could not find container \"d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b\": container with ID starting with d737efe3cc4fccaa749b50a02c084049a21991026a42dc9422e338d643c4140b not found: ID does not exist"
Jul 29 11:25:53.882426 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.882405    2551 scope.go:117] "RemoveContainer" containerID="bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"
Jul 29 11:25:53.882663 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:25:53.882645    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5\": container with ID starting with bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5 not found: ID does not exist" containerID="bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"
Jul 29 11:25:53.882711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.882670    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5"} err="failed to get container status \"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5\": rpc error: code = NotFound desc = could not find container \"bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5\": container with ID starting with bd9aeec250d597c63c023f054a51bd9bd3e28cbd035b5d05e6c723a641cc3bc5 not found: ID does not exist"
Jul 29 11:25:53.901864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.901830    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:53.903643 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:53.903621    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-541eb7-predictor-66c5b888ff-ks642"]
Jul 29 11:25:54.728936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:54.728896    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" path="/var/lib/kubelet/pods/7c4aaf59-5591-4ec1-88e3-706b43ee9194/volumes"
Jul 29 11:25:58.371565 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.371525    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:25:58.372000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.371982    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.372089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.372002    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.372089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.372059    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.372089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.372066    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.372246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.372120    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.372246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.372139    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4aaf59-5591-4ec1-88e3-706b43ee9194" containerName="storage-initializer"
Jul 29 11:25:58.376652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.376631    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.379225 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379196    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 29 11:25:58.379352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379202    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:25:58.379352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379247    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 29 11:25:58.379352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379202    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:25:58.379515 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379470    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:25:58.379515 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.379485    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:25:58.380200 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.380182    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:25:58.385592 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.385564    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:25:58.473016 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.472991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.473145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.473030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.473145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.473048    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvlm\" (UniqueName: \"kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.473145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.473069    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.473269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.473196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.573643 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.573615    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.573757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.573657    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.573757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.573686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.573757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.573707    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvlm\" (UniqueName: \"kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.573932 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.573809    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.574120 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.574097    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.574573 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.574547    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.574665 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.574557    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.576439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.576415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.581846 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.581811    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvlm\" (UniqueName: \"kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-7jnn9\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.688606 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.688522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:25:58.823246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.823222    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:25:58.825814 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:25:58.825786    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fb6c93_858c_4a5a_ada0_89e921024ab1.slice/crio-4daa49386e68ae524d594c4f9582a0592480f56a02640f3a171f8b4637fdb0af WatchSource:0}: Error finding container 4daa49386e68ae524d594c4f9582a0592480f56a02640f3a171f8b4637fdb0af: Status 404 returned error can't find the container with id 4daa49386e68ae524d594c4f9582a0592480f56a02640f3a171f8b4637fdb0af
Jul 29 11:25:58.891614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.891581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerStarted","Data":"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"}
Jul 29 11:25:58.891711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:25:58.891643    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerStarted","Data":"4daa49386e68ae524d594c4f9582a0592480f56a02640f3a171f8b4637fdb0af"}
Jul 29 11:26:02.910518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:02.910492    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerID="623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a" exitCode=0
Jul 29 11:26:02.910886 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:02.910571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerDied","Data":"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"}
Jul 29 11:26:02.911705 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:02.911687    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:26:23.040298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:23.040266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerStarted","Data":"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"}
Jul 29 11:26:23.040705 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:23.040328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerStarted","Data":"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"}
Jul 29 11:26:23.040705 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:23.040520    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:26:23.063086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:23.063036    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podStartSLOduration=1.7294295910000002 podStartE2EDuration="25.063022213s" totalImagesPullingTime="19.771972331s" totalInitContainerRuntime="3.561620291s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:26:23.061586038 +0000 UTC m=+3434.963814431" watchObservedRunningTime="2026-07-29 11:26:23.063022213 +0000 UTC m=+3434.965250609"
Jul 29 11:26:24.044686 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:24.044653    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:26:24.045774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:24.045746    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:26:25.048856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:25.048810    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:26:30.053783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:30.053750    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:26:30.054428 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:30.054398    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:26:40.054511 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:40.054468    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:26:50.055093 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:26:50.055053    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:00.055052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:00.055005    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:10.054436 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:10.054391    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:20.054992 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:20.054946    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:30.055001 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:30.054911    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:40.055615 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:40.055582    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:27:48.530672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:48.530641    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:27:48.531227 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:48.530917    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" containerID="cri-o://797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9" gracePeriod=30
Jul 29 11:27:48.531227 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:48.530957    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kube-rbac-proxy" containerID="cri-o://2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7" gracePeriod=30
Jul 29 11:27:49.420879 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:49.420842    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerID="2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7" exitCode=2
Jul 29 11:27:49.421059 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:49.420913    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerDied","Data":"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"}
Jul 29 11:27:50.049997 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:50.049955    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 29 11:27:50.054310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:50.054282    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 29 11:27:52.283929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.283901    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:27:52.421277 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421200    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location\") pod \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") "
Jul 29 11:27:52.421277 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421247    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls\") pod \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") "
Jul 29 11:27:52.421277 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421268    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config\") pod \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") "
Jul 29 11:27:52.421604 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421473    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert\") pod \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") "
Jul 29 11:27:52.421604 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421517    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gvlm\" (UniqueName: \"kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm\") pod \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\" (UID: \"d4fb6c93-858c-4a5a-ada0-89e921024ab1\") "
Jul 29 11:27:52.421604 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421528    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4fb6c93-858c-4a5a-ada0-89e921024ab1" (UID: "d4fb6c93-858c-4a5a-ada0-89e921024ab1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:27:52.421754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421645    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4fb6c93-858c-4a5a-ada0-89e921024ab1" (UID: "d4fb6c93-858c-4a5a-ada0-89e921024ab1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:27:52.421810 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421783    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4fb6c93-858c-4a5a-ada0-89e921024ab1" (UID: "d4fb6c93-858c-4a5a-ada0-89e921024ab1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:27:52.421810 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421793    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:27:52.421900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.421813    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:27:52.423404 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.423380    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4fb6c93-858c-4a5a-ada0-89e921024ab1" (UID: "d4fb6c93-858c-4a5a-ada0-89e921024ab1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:27:52.423558 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.423533    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm" (OuterVolumeSpecName: "kube-api-access-2gvlm") pod "d4fb6c93-858c-4a5a-ada0-89e921024ab1" (UID: "d4fb6c93-858c-4a5a-ada0-89e921024ab1"). InnerVolumeSpecName "kube-api-access-2gvlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:27:52.436390 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.436361    2551 generic.go:356] "Generic (PLEG): container finished" podID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerID="797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9" exitCode=0
Jul 29 11:27:52.436517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.436397    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerDied","Data":"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"}
Jul 29 11:27:52.436517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.436426    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"
Jul 29 11:27:52.436517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.436492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9" event={"ID":"d4fb6c93-858c-4a5a-ada0-89e921024ab1","Type":"ContainerDied","Data":"4daa49386e68ae524d594c4f9582a0592480f56a02640f3a171f8b4637fdb0af"}
Jul 29 11:27:52.436677 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.436535    2551 scope.go:117] "RemoveContainer" containerID="2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"
Jul 29 11:27:52.444931 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.444914    2551 scope.go:117] "RemoveContainer" containerID="797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"
Jul 29 11:27:52.452196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.452180    2551 scope.go:117] "RemoveContainer" containerID="623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"
Jul 29 11:27:52.457990 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.457965    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:27:52.459753 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.459731    2551 scope.go:117] "RemoveContainer" containerID="2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"
Jul 29 11:27:52.460002 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:27:52.459978    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7\": container with ID starting with 2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7 not found: ID does not exist" containerID="2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"
Jul 29 11:27:52.460104 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.460013    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7"} err="failed to get container status \"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7\": rpc error: code = NotFound desc = could not find container \"2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7\": container with ID starting with 2608fd8ad5789564fc1eb27f1d766fdeea3f9dca5254b8d3c12bb9e3089228d7 not found: ID does not exist"
Jul 29 11:27:52.460104 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.460036    2551 scope.go:117] "RemoveContainer" containerID="797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"
Jul 29 11:27:52.460422 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:27:52.460341    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9\": container with ID starting with 797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9 not found: ID does not exist" containerID="797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"
Jul 29 11:27:52.460422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.460370    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9"} err="failed to get container status \"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9\": rpc error: code = NotFound desc = could not find container \"797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9\": container with ID starting with 797117acfa0ccd2aad14fd0e2a5b87be37464a65635dc16c707497fb259989c9 not found: ID does not exist"
Jul 29 11:27:52.460422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.460390    2551 scope.go:117] "RemoveContainer" containerID="623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"
Jul 29 11:27:52.460704 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:27:52.460682    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a\": container with ID starting with 623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a not found: ID does not exist" containerID="623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"
Jul 29 11:27:52.460750 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.460713    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a"} err="failed to get container status \"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a\": rpc error: code = NotFound desc = could not find container \"623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a\": container with ID starting with 623d33c74fc31e3e3a8d430be58fc8df44b55ee8881f59abf01eec5c8a78648a not found: ID does not exist"
Jul 29 11:27:52.462211 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.462193    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-7jnn9"]
Jul 29 11:27:52.522800 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.522775    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4fb6c93-858c-4a5a-ada0-89e921024ab1-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:27:52.522800 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.522797    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvlm\" (UniqueName: \"kubernetes.io/projected/d4fb6c93-858c-4a5a-ada0-89e921024ab1-kube-api-access-2gvlm\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:27:52.522921 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.522811    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4fb6c93-858c-4a5a-ada0-89e921024ab1-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:27:52.728674 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:52.728645    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" path="/var/lib/kubelet/pods/d4fb6c93-858c-4a5a-ada0-89e921024ab1/volumes"
Jul 29 11:27:53.702910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.702879    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703203    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="storage-initializer"
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703216    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="storage-initializer"
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703224    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kube-rbac-proxy"
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703230    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kube-rbac-proxy"
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703252    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container"
Jul 29 11:27:53.703286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703257    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container"
Jul 29 11:27:53.703481 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703309    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kserve-container"
Jul 29 11:27:53.703481 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.703318    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fb6c93-858c-4a5a-ada0-89e921024ab1" containerName="kube-rbac-proxy"
Jul 29 11:27:53.705497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.705481    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.707953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.707933    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 29 11:27:53.708082 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.707973    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:27:53.708082 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.707977    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:27:53.708082 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.708047    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:27:53.708082 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.708049    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:27:53.708290 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.708184    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 29 11:27:53.708889 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.708872    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:27:53.715914 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.715895    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:27:53.832103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.832082    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.832215 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.832113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.832215 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.832177    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.832323 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.832222    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.832323 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.832245    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vw7\" (UniqueName: \"kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.932689 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.932663    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.932786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.932697    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.932786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.932723    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.932786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.932766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.932908 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.932799    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vw7\" (UniqueName: \"kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.933175 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.933155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.933506 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.933477    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.933506 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.933496    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.935379 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.935353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:53.939838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:53.939812    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vw7\" (UniqueName: \"kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7\") pod \"isvc-predictive-xgboost-predictor-695f978978-tg8gn\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:54.016671 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:54.016614    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:54.146295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:54.146263    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:27:54.149420 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:27:54.149395    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d26eab_e4bc_4d5e_a4bd_ca434b843c34.slice/crio-4b993dfa998055c3af98bfb5993ab98350b3fbe0c61818f57c91abcbfd6e6cb7 WatchSource:0}: Error finding container 4b993dfa998055c3af98bfb5993ab98350b3fbe0c61818f57c91abcbfd6e6cb7: Status 404 returned error can't find the container with id 4b993dfa998055c3af98bfb5993ab98350b3fbe0c61818f57c91abcbfd6e6cb7
Jul 29 11:27:54.447367 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:54.447272    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerStarted","Data":"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"}
Jul 29 11:27:54.447367 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:54.447342    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerStarted","Data":"4b993dfa998055c3af98bfb5993ab98350b3fbe0c61818f57c91abcbfd6e6cb7"}
Jul 29 11:27:58.470318 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:58.470276    2551 generic.go:356] "Generic (PLEG): container finished" podID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerID="d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239" exitCode=0
Jul 29 11:27:58.470708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:58.470367    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerDied","Data":"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"}
Jul 29 11:27:59.477568 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:59.477529    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerStarted","Data":"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"}
Jul 29 11:27:59.477984 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:59.477592    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerStarted","Data":"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"}
Jul 29 11:27:59.477984 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:59.477775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:27:59.496269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:27:59.496229    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podStartSLOduration=2.836393557 podStartE2EDuration="6.496214861s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659821304s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:27:59.495445125 +0000 UTC m=+3531.397673552" watchObservedRunningTime="2026-07-29 11:27:59.496214861 +0000 UTC m=+3531.398443256"
Jul 29 11:28:00.483715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:00.483678    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:28:00.484898 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:00.484867    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:01.488902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:01.488862    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:06.493667 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:06.493639    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:28:06.494280 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:06.494246    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:16.494902 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:16.494854    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:26.494887 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:26.494849    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:36.494733 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:36.494694    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:46.494074 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:46.494029    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:28:56.494232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:28:56.494178    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:29:06.495284 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:06.495258    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:29:08.870499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:08.870468    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:29:08.880113 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:08.880096    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:29:13.763639 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:13.763583    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:29:13.764158 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:13.763909    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" containerID="cri-o://ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4" gracePeriod=30
Jul 29 11:29:13.764158 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:13.763948    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kube-rbac-proxy" containerID="cri-o://63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123" gracePeriod=30
Jul 29 11:29:14.801106 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:14.801065    2551 generic.go:356] "Generic (PLEG): container finished" podID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerID="63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123" exitCode=2
Jul 29 11:29:14.801494 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:14.801150    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerDied","Data":"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"}
Jul 29 11:29:16.489690 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:16.489607    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 29 11:29:16.495046 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:16.495022    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 29 11:29:17.507955 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.507932    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:29:17.531673 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.531648    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config\") pod \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") "
Jul 29 11:29:17.531781 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.531684    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert\") pod \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") "
Jul 29 11:29:17.531781 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.531741    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls\") pod \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") "
Jul 29 11:29:17.531781 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.531769    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vw7\" (UniqueName: \"kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7\") pod \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") "
Jul 29 11:29:17.531962 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.531799    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location\") pod \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\" (UID: \"13d26eab-e4bc-4d5e-a4bd-ca434b843c34\") "
Jul 29 11:29:17.532034 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.532006    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13d26eab-e4bc-4d5e-a4bd-ca434b843c34" (UID: "13d26eab-e4bc-4d5e-a4bd-ca434b843c34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:29:17.532094 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.532046    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13d26eab-e4bc-4d5e-a4bd-ca434b843c34" (UID: "13d26eab-e4bc-4d5e-a4bd-ca434b843c34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:29:17.532296 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.532255    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13d26eab-e4bc-4d5e-a4bd-ca434b843c34" (UID: "13d26eab-e4bc-4d5e-a4bd-ca434b843c34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:29:17.533741 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.533714    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13d26eab-e4bc-4d5e-a4bd-ca434b843c34" (UID: "13d26eab-e4bc-4d5e-a4bd-ca434b843c34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:29:17.533994 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.533973    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7" (OuterVolumeSpecName: "kube-api-access-m7vw7") pod "13d26eab-e4bc-4d5e-a4bd-ca434b843c34" (UID: "13d26eab-e4bc-4d5e-a4bd-ca434b843c34"). InnerVolumeSpecName "kube-api-access-m7vw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:29:17.633141 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.633106    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:29:17.633235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.633145    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:29:17.633235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.633156    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:29:17.633235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.633165    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vw7\" (UniqueName: \"kubernetes.io/projected/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kube-api-access-m7vw7\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:29:17.633235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.633174    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13d26eab-e4bc-4d5e-a4bd-ca434b843c34-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:29:17.815210 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.815141    2551 generic.go:356] "Generic (PLEG): container finished" podID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerID="ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4" exitCode=0
Jul 29 11:29:17.815315 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.815221    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"
Jul 29 11:29:17.815315 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.815227    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerDied","Data":"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"}
Jul 29 11:29:17.815315 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.815295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn" event={"ID":"13d26eab-e4bc-4d5e-a4bd-ca434b843c34","Type":"ContainerDied","Data":"4b993dfa998055c3af98bfb5993ab98350b3fbe0c61818f57c91abcbfd6e6cb7"}
Jul 29 11:29:17.815423 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.815337    2551 scope.go:117] "RemoveContainer" containerID="63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"
Jul 29 11:29:17.824766 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.824746    2551 scope.go:117] "RemoveContainer" containerID="ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"
Jul 29 11:29:17.832673 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.832652    2551 scope.go:117] "RemoveContainer" containerID="d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"
Jul 29 11:29:17.838543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.838514    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:29:17.840880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.840858    2551 scope.go:117] "RemoveContainer" containerID="63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"
Jul 29 11:29:17.841263 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:29:17.841242    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123\": container with ID starting with 63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123 not found: ID does not exist" containerID="63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"
Jul 29 11:29:17.841352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.841276    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123"} err="failed to get container status \"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123\": rpc error: code = NotFound desc = could not find container \"63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123\": container with ID starting with 63cf100bfd6051c548161fb455507fe5a8dc1b4f9d7bd3c2fa65630959462123 not found: ID does not exist"
Jul 29 11:29:17.841352 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.841302    2551 scope.go:117] "RemoveContainer" containerID="ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"
Jul 29 11:29:17.841613 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:29:17.841593    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4\": container with ID starting with ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4 not found: ID does not exist" containerID="ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"
Jul 29 11:29:17.841662 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.841620    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4"} err="failed to get container status \"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4\": rpc error: code = NotFound desc = could not find container \"ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4\": container with ID starting with ab90fc943a94ce8ed76d6020430e7d0f54a6674b8c7fc5cc2a08ebc6e1b9a3c4 not found: ID does not exist"
Jul 29 11:29:17.841662 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.841637    2551 scope.go:117] "RemoveContainer" containerID="d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"
Jul 29 11:29:17.841921 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:29:17.841903    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239\": container with ID starting with d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239 not found: ID does not exist" containerID="d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"
Jul 29 11:29:17.841993 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.841930    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239"} err="failed to get container status \"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239\": rpc error: code = NotFound desc = could not find container \"d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239\": container with ID starting with d674e0e0ab6be535404f2236e86866c7adf2711b7ef1da63a5b73e15338c4239 not found: ID does not exist"
Jul 29 11:29:17.842300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:17.842282    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-tg8gn"]
Jul 29 11:29:18.729617 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.729583    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" path="/var/lib/kubelet/pods/13d26eab-e4bc-4d5e-a4bd-ca434b843c34/volumes"
Jul 29 11:29:18.938359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938321    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:29:18.938717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938700    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="storage-initializer"
Jul 29 11:29:18.938802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938720    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="storage-initializer"
Jul 29 11:29:18.938802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938737    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kube-rbac-proxy"
Jul 29 11:29:18.938802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938745    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kube-rbac-proxy"
Jul 29 11:29:18.938802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938764    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container"
Jul 29 11:29:18.938802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938773    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container"
Jul 29 11:29:18.939066 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938864    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kube-rbac-proxy"
Jul 29 11:29:18.939066 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.938876    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d26eab-e4bc-4d5e-a4bd-ca434b843c34" containerName="kserve-container"
Jul 29 11:29:18.943371 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.943350    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:18.945984 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.945929    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:29:18.945984 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.945957    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 29 11:29:18.946201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.946046    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:29:18.946201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.946069    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:29:18.946900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.946883    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 29 11:29:18.947082 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.947070    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:29:18.947181 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.947167    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:29:18.951705 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:18.951684    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:29:19.045981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.045915    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlpm\" (UniqueName: \"kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.045981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.045955    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.046151 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.045983    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.046151 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.046012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.046151 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.046036    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlpm\" (UniqueName: \"kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147332 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147247    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147332 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147274    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147332 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147493 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147493 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:29:19.147347    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 29 11:29:19.147493 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:29:19.147416    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls podName:11703b8e-36c1-4a82-a7f4-efb9cd9d8893 nodeName:}" failed. No retries permitted until 2026-07-29 11:29:19.647395344 +0000 UTC m=+3611.549623720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 29 11:29:19.147734 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147715    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.147985 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.147965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.148112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.148092    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.156294 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.156272    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlpm\" (UniqueName: \"kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.651367 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.651340    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.653717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.653697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.854776 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.854745    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:19.979187 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:19.979021    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:29:19.981551 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:29:19.981522    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11703b8e_36c1_4a82_a7f4_efb9cd9d8893.slice/crio-1b4c839ff5ba67bf0a6b9c71223ca269dc10dcaaa16a3b00a51da2a2cf518abb WatchSource:0}: Error finding container 1b4c839ff5ba67bf0a6b9c71223ca269dc10dcaaa16a3b00a51da2a2cf518abb: Status 404 returned error can't find the container with id 1b4c839ff5ba67bf0a6b9c71223ca269dc10dcaaa16a3b00a51da2a2cf518abb
Jul 29 11:29:20.829751 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:20.829715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerStarted","Data":"e783fa1225d8c523a7011264071d1c645373d2efc57038e2a8552afe48d00fc3"}
Jul 29 11:29:20.829933 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:20.829767    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerStarted","Data":"1b4c839ff5ba67bf0a6b9c71223ca269dc10dcaaa16a3b00a51da2a2cf518abb"}
Jul 29 11:29:24.849010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:24.848974    2551 generic.go:356] "Generic (PLEG): container finished" podID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerID="e783fa1225d8c523a7011264071d1c645373d2efc57038e2a8552afe48d00fc3" exitCode=0
Jul 29 11:29:24.849513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:24.849050    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerDied","Data":"e783fa1225d8c523a7011264071d1c645373d2efc57038e2a8552afe48d00fc3"}
Jul 29 11:29:25.855571 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:25.855534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerStarted","Data":"51bd21d70309ab397ad298a7211fd5ce7a0a47597ccb1f4d08e7310a4a8e39e6"}
Jul 29 11:29:25.855953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:25.855601    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerStarted","Data":"ad2f3687124e61937c494dd8bdd22e71edd68e4c705b5c3e109bbc6700fb1a82"}
Jul 29 11:29:25.855953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:25.855799    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:25.875184 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:25.875116    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podStartSLOduration=4.000991246 podStartE2EDuration="7.875103551s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874112305s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:29:25.873567605 +0000 UTC m=+3617.775796001" watchObservedRunningTime="2026-07-29 11:29:25.875103551 +0000 UTC m=+3617.777331947"
Jul 29 11:29:26.860249 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:26.860217    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:26.861467 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:26.861441    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:29:27.864372 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:27.864329    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:29:32.868596 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:32.868567    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:29:32.869179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:32.869146    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:29:42.869384 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:42.869337    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:29:52.869010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:29:52.868969    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:30:02.869344 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:02.869302    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:30:12.869011 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:12.868971    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:30:22.869160 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:22.869099    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:30:32.870321 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:32.870228    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:30:39.071573 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:39.071540    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:30:39.072067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:39.071860    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" containerID="cri-o://ad2f3687124e61937c494dd8bdd22e71edd68e4c705b5c3e109bbc6700fb1a82" gracePeriod=30
Jul 29 11:30:39.072067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:39.071858    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kube-rbac-proxy" containerID="cri-o://51bd21d70309ab397ad298a7211fd5ce7a0a47597ccb1f4d08e7310a4a8e39e6" gracePeriod=30
Jul 29 11:30:40.164793 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:40.164761    2551 generic.go:356] "Generic (PLEG): container finished" podID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerID="51bd21d70309ab397ad298a7211fd5ce7a0a47597ccb1f4d08e7310a4a8e39e6" exitCode=2
Jul 29 11:30:40.165203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:40.164833    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerDied","Data":"51bd21d70309ab397ad298a7211fd5ce7a0a47597ccb1f4d08e7310a4a8e39e6"}
Jul 29 11:30:42.865085 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:42.865050    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Jul 29 11:30:42.869413 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:42.869384    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 29 11:30:43.179656 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.179587    2551 generic.go:356] "Generic (PLEG): container finished" podID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerID="ad2f3687124e61937c494dd8bdd22e71edd68e4c705b5c3e109bbc6700fb1a82" exitCode=0
Jul 29 11:30:43.179656 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.179641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerDied","Data":"ad2f3687124e61937c494dd8bdd22e71edd68e4c705b5c3e109bbc6700fb1a82"}
Jul 29 11:30:43.418021 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.417997    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:30:43.454003 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.453973    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") pod \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") "
Jul 29 11:30:43.454157 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454019    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config\") pod \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") "
Jul 29 11:30:43.454157 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert\") pod \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") "
Jul 29 11:30:43.454157 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454089    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location\") pod \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") "
Jul 29 11:30:43.454157 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454114    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wlpm\" (UniqueName: \"kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm\") pod \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\" (UID: \"11703b8e-36c1-4a82-a7f4-efb9cd9d8893\") "
Jul 29 11:30:43.454484 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454457    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "11703b8e-36c1-4a82-a7f4-efb9cd9d8893" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:30:43.454572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454486    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11703b8e-36c1-4a82-a7f4-efb9cd9d8893" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:30:43.454572 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.454504    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "11703b8e-36c1-4a82-a7f4-efb9cd9d8893" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:30:43.455927 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.455903    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11703b8e-36c1-4a82-a7f4-efb9cd9d8893" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:30:43.456161 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.456140    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm" (OuterVolumeSpecName: "kube-api-access-7wlpm") pod "11703b8e-36c1-4a82-a7f4-efb9cd9d8893" (UID: "11703b8e-36c1-4a82-a7f4-efb9cd9d8893"). InnerVolumeSpecName "kube-api-access-7wlpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:30:43.555744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.555716    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:30:43.555744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.555743    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:30:43.555881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.555758    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:30:43.555881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.555772    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:30:43.555881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:43.555785    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wlpm\" (UniqueName: \"kubernetes.io/projected/11703b8e-36c1-4a82-a7f4-efb9cd9d8893-kube-api-access-7wlpm\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:30:44.186326 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.186285    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt" event={"ID":"11703b8e-36c1-4a82-a7f4-efb9cd9d8893","Type":"ContainerDied","Data":"1b4c839ff5ba67bf0a6b9c71223ca269dc10dcaaa16a3b00a51da2a2cf518abb"}
Jul 29 11:30:44.186737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.186382    2551 scope.go:117] "RemoveContainer" containerID="51bd21d70309ab397ad298a7211fd5ce7a0a47597ccb1f4d08e7310a4a8e39e6"
Jul 29 11:30:44.186737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.186300    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"
Jul 29 11:30:44.199766 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.199745    2551 scope.go:117] "RemoveContainer" containerID="ad2f3687124e61937c494dd8bdd22e71edd68e4c705b5c3e109bbc6700fb1a82"
Jul 29 11:30:44.207540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.207523    2551 scope.go:117] "RemoveContainer" containerID="e783fa1225d8c523a7011264071d1c645373d2efc57038e2a8552afe48d00fc3"
Jul 29 11:30:44.217004 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.216985    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:30:44.221850 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.221826    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-6rrbt"]
Jul 29 11:30:44.244179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244160    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:30:44.244477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244465    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="storage-initializer"
Jul 29 11:30:44.244519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244479    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="storage-initializer"
Jul 29 11:30:44.244519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244487    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container"
Jul 29 11:30:44.244519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244492    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container"
Jul 29 11:30:44.244519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244499    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kube-rbac-proxy"
Jul 29 11:30:44.244519 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244509    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kube-rbac-proxy"
Jul 29 11:30:44.244671 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244567    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kserve-container"
Jul 29 11:30:44.244671 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.244574    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" containerName="kube-rbac-proxy"
Jul 29 11:30:44.248898 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.248883    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.251477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251459    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:30:44.251569 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251493    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 29 11:30:44.251569 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251464    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 29 11:30:44.251569 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251532    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:30:44.251569 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251546    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:30:44.251806 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251645    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:30:44.251850 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.251839    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:30:44.258193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.258174    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:30:44.360815 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.360790    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.360927 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.360819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zszg\" (UniqueName: \"kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.360927 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.360863    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.360927 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.360916    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.361064 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.360936    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.461364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461345    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.461481 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461373    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zszg\" (UniqueName: \"kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.461481 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461430    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.461605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461493    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.461605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461518    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.462000 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.461953    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.462305 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.462284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.462366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.462284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.463771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.463751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.471203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.471178    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zszg\" (UniqueName: \"kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.558737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.558709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:44.684498 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.684467    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:30:44.686628 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:30:44.686592    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4013ec_654d_4483_a294_25a689b26a48.slice/crio-5cf5dbd29611eef70c7bf832a19e07d48746103d181ef3863a0b372f8fb9add0 WatchSource:0}: Error finding container 5cf5dbd29611eef70c7bf832a19e07d48746103d181ef3863a0b372f8fb9add0: Status 404 returned error can't find the container with id 5cf5dbd29611eef70c7bf832a19e07d48746103d181ef3863a0b372f8fb9add0
Jul 29 11:30:44.729494 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:44.729471    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11703b8e-36c1-4a82-a7f4-efb9cd9d8893" path="/var/lib/kubelet/pods/11703b8e-36c1-4a82-a7f4-efb9cd9d8893/volumes"
Jul 29 11:30:45.192444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:45.192347    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerStarted","Data":"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"}
Jul 29 11:30:45.192444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:45.192420    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerStarted","Data":"5cf5dbd29611eef70c7bf832a19e07d48746103d181ef3863a0b372f8fb9add0"}
Jul 29 11:30:49.212443 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:49.212416    2551 generic.go:356] "Generic (PLEG): container finished" podID="3d4013ec-654d-4483-a294-25a689b26a48" containerID="3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8" exitCode=0
Jul 29 11:30:49.212786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:49.212459    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerDied","Data":"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"}
Jul 29 11:30:50.220981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:50.220949    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerStarted","Data":"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"}
Jul 29 11:30:50.221459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:50.221009    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerStarted","Data":"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"}
Jul 29 11:30:50.221459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:50.221259    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:50.240607 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:50.240547    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podStartSLOduration=2.272322011 podStartE2EDuration="6.240527949s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.968205938s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:30:50.239257196 +0000 UTC m=+3702.141485594" watchObservedRunningTime="2026-07-29 11:30:50.240527949 +0000 UTC m=+3702.142756347"
Jul 29 11:30:51.225111 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:51.225079    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:30:57.234819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:30:57.234785    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:31:27.236047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:31:27.236011    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 29 11:31:37.236207 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:31:37.236166    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 29 11:31:47.235311 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:31:47.235275    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 29 11:31:57.236035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:31:57.235988    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 29 11:32:00.789844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:00.789748    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:32:05.224593 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:05.224558    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:32:05.225017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:05.224804    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container" containerID="cri-o://8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b" gracePeriod=30
Jul 29 11:32:05.225017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:05.224848    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kube-rbac-proxy" containerID="cri-o://f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a" gracePeriod=30
Jul 29 11:32:05.553672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:05.553611    2551 generic.go:356] "Generic (PLEG): container finished" podID="3d4013ec-654d-4483-a294-25a689b26a48" containerID="f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a" exitCode=2
Jul 29 11:32:05.553672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:05.553671    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerDied","Data":"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"}
Jul 29 11:32:07.230350 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:07.230312    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 29 11:32:08.966311 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:08.966283    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:32:09.063646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063587    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert\") pod \"3d4013ec-654d-4483-a294-25a689b26a48\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") "
Jul 29 11:32:09.063646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063630    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config\") pod \"3d4013ec-654d-4483-a294-25a689b26a48\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") "
Jul 29 11:32:09.063818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063658    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location\") pod \"3d4013ec-654d-4483-a294-25a689b26a48\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") "
Jul 29 11:32:09.063818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063686    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zszg\" (UniqueName: \"kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg\") pod \"3d4013ec-654d-4483-a294-25a689b26a48\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") "
Jul 29 11:32:09.063818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063758    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls\") pod \"3d4013ec-654d-4483-a294-25a689b26a48\" (UID: \"3d4013ec-654d-4483-a294-25a689b26a48\") "
Jul 29 11:32:09.064014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063988    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d4013ec-654d-4483-a294-25a689b26a48" (UID: "3d4013ec-654d-4483-a294-25a689b26a48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:32:09.064083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063998    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d4013ec-654d-4483-a294-25a689b26a48" (UID: "3d4013ec-654d-4483-a294-25a689b26a48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:32:09.064083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.063995    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d4013ec-654d-4483-a294-25a689b26a48" (UID: "3d4013ec-654d-4483-a294-25a689b26a48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:32:09.065877 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.065841    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d4013ec-654d-4483-a294-25a689b26a48" (UID: "3d4013ec-654d-4483-a294-25a689b26a48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:32:09.065877 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.065851    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg" (OuterVolumeSpecName: "kube-api-access-5zszg") pod "3d4013ec-654d-4483-a294-25a689b26a48" (UID: "3d4013ec-654d-4483-a294-25a689b26a48"). InnerVolumeSpecName "kube-api-access-5zszg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:32:09.164242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.164216    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d4013ec-654d-4483-a294-25a689b26a48-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:32:09.164242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.164239    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zszg\" (UniqueName: \"kubernetes.io/projected/3d4013ec-654d-4483-a294-25a689b26a48-kube-api-access-5zszg\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:32:09.164368 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.164250    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d4013ec-654d-4483-a294-25a689b26a48-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:32:09.164368 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.164260    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:32:09.164368 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.164269    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d4013ec-654d-4483-a294-25a689b26a48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:32:09.573077 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.573044    2551 generic.go:356] "Generic (PLEG): container finished" podID="3d4013ec-654d-4483-a294-25a689b26a48" containerID="8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b" exitCode=0
Jul 29 11:32:09.573219 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.573114    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"
Jul 29 11:32:09.573219 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.573144    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerDied","Data":"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"}
Jul 29 11:32:09.573219 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.573202    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5" event={"ID":"3d4013ec-654d-4483-a294-25a689b26a48","Type":"ContainerDied","Data":"5cf5dbd29611eef70c7bf832a19e07d48746103d181ef3863a0b372f8fb9add0"}
Jul 29 11:32:09.573340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.573234    2551 scope.go:117] "RemoveContainer" containerID="f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"
Jul 29 11:32:09.582844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.582824    2551 scope.go:117] "RemoveContainer" containerID="8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"
Jul 29 11:32:09.591910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.591887    2551 scope.go:117] "RemoveContainer" containerID="3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"
Jul 29 11:32:09.595193 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.595167    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:32:09.599121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.599094    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlv5"]
Jul 29 11:32:09.604010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.603992    2551 scope.go:117] "RemoveContainer" containerID="f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"
Jul 29 11:32:09.604293 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:32:09.604272    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a\": container with ID starting with f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a not found: ID does not exist" containerID="f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"
Jul 29 11:32:09.604375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.604299    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a"} err="failed to get container status \"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a\": rpc error: code = NotFound desc = could not find container \"f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a\": container with ID starting with f52499a40c52c60e228f7847a2756adc14552dcc83434e6c97e5b46c1da4680a not found: ID does not exist"
Jul 29 11:32:09.604375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.604318    2551 scope.go:117] "RemoveContainer" containerID="8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"
Jul 29 11:32:09.604575 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:32:09.604553    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b\": container with ID starting with 8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b not found: ID does not exist" containerID="8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"
Jul 29 11:32:09.604644 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.604585    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b"} err="failed to get container status \"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b\": rpc error: code = NotFound desc = could not find container \"8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b\": container with ID starting with 8f9576e3321c94f2285c99cc4198840758a9922374cf7f003a465c9a600f655b not found: ID does not exist"
Jul 29 11:32:09.604644 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.604610    2551 scope.go:117] "RemoveContainer" containerID="3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"
Jul 29 11:32:09.604869 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:32:09.604850    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8\": container with ID starting with 3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8 not found: ID does not exist" containerID="3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"
Jul 29 11:32:09.604932 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:09.604874    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8"} err="failed to get container status \"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8\": rpc error: code = NotFound desc = could not find container \"3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8\": container with ID starting with 3708eddd09e8b2a32c599c6737114666c8f562a3ce86c79581f2f17c8c7faec8 not found: ID does not exist"
Jul 29 11:32:10.408792 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.408757    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409090    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409101    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409111    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kube-rbac-proxy"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409116    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kube-rbac-proxy"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409143    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="storage-initializer"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409149    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="storage-initializer"
Jul 29 11:32:10.409199 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409200    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kube-rbac-proxy"
Jul 29 11:32:10.409429 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.409207    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4013ec-654d-4483-a294-25a689b26a48" containerName="kserve-container"
Jul 29 11:32:10.413867 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.413849    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.416492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.416462    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 29 11:32:10.416612 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.416471    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:32:10.416612 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.416472    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 29 11:32:10.416612 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.416534    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:32:10.417562 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.417545    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:32:10.417660 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.417577    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:32:10.417660 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.417607    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:32:10.423041 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.423021    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:32:10.474116 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.474095    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.474232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.474159    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.474232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.474183    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9r8\" (UniqueName: \"kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.474232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.474209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.474369 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.474288    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.574678 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.574658    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.574773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.574698    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9r8\" (UniqueName: \"kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.574773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.574737    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.574886 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.574785    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.574886 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.574827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.575015 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:32:10.574996    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 29 11:32:10.575167 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:32:10.575058    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls podName:baf4df9f-9a99-4d0e-ac1c-88310aa6fbef nodeName:}" failed. No retries permitted until 2026-07-29 11:32:11.075040172 +0000 UTC m=+3782.977268550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 29 11:32:10.575258 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.575198    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.575440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.575419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.575487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.575419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.585554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.585536    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9r8\" (UniqueName: \"kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:10.729044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:10.729012    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d4013ec-654d-4483-a294-25a689b26a48" path="/var/lib/kubelet/pods/3d4013ec-654d-4483-a294-25a689b26a48/volumes"
Jul 29 11:32:11.078362 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.078282    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:11.080736 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.080703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:11.326179 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.326150    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:11.462480 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.462457    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:32:11.464734 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:32:11.464708    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf4df9f_9a99_4d0e_ac1c_88310aa6fbef.slice/crio-b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc WatchSource:0}: Error finding container b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc: Status 404 returned error can't find the container with id b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc
Jul 29 11:32:11.584330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.584256    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerStarted","Data":"6241e0b2d1b541124de6f56abf9aa0c307bb4789bd1a0bd151ff0bfe0abccb57"}
Jul 29 11:32:11.584454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:11.584325    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerStarted","Data":"b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc"}
Jul 29 11:32:15.608256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:15.608223    2551 generic.go:356] "Generic (PLEG): container finished" podID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerID="6241e0b2d1b541124de6f56abf9aa0c307bb4789bd1a0bd151ff0bfe0abccb57" exitCode=0
Jul 29 11:32:15.608623 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:15.608264    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerDied","Data":"6241e0b2d1b541124de6f56abf9aa0c307bb4789bd1a0bd151ff0bfe0abccb57"}
Jul 29 11:32:16.614958 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:16.614924    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerStarted","Data":"08ae57a2472dbdc090b02010c96238224d3bb6e97417c9de93a32302a92c191a"}
Jul 29 11:32:16.615341 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:16.614986    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerStarted","Data":"b8df54164b91c98ee64268fe7261f1d133b38e459ec585c3fee7a615d4e34fbf"}
Jul 29 11:32:16.615341 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:16.615221    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:16.645022 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:16.644973    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podStartSLOduration=2.981112325 podStartE2EDuration="6.64495806s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663845735s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:32:16.643351833 +0000 UTC m=+3788.545580233" watchObservedRunningTime="2026-07-29 11:32:16.64495806 +0000 UTC m=+3788.547186457"
Jul 29 11:32:17.619832 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:17.619797    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:23.628354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:23.628325    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:32:53.629652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:32:53.629614    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.60:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 29 11:33:03.629520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:03.629474    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.60:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 29 11:33:13.629456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:13.629410    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.60:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 29 11:33:23.686827 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:23.686793    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:33:30.500597 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:30.500510    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:33:30.501079 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:30.500785    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container" containerID="cri-o://b8df54164b91c98ee64268fe7261f1d133b38e459ec585c3fee7a615d4e34fbf" gracePeriod=30
Jul 29 11:33:30.501079 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:30.500829    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kube-rbac-proxy" containerID="cri-o://08ae57a2472dbdc090b02010c96238224d3bb6e97417c9de93a32302a92c191a" gracePeriod=30
Jul 29 11:33:30.913316 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:30.913235    2551 generic.go:356] "Generic (PLEG): container finished" podID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerID="08ae57a2472dbdc090b02010c96238224d3bb6e97417c9de93a32302a92c191a" exitCode=2
Jul 29 11:33:30.913444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:30.913316    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerDied","Data":"08ae57a2472dbdc090b02010c96238224d3bb6e97417c9de93a32302a92c191a"}
Jul 29 11:33:33.624029 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:33.623984    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 29 11:33:33.629456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:33.629430    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.60:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 29 11:33:34.933203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:34.933175    2551 generic.go:356] "Generic (PLEG): container finished" podID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerID="b8df54164b91c98ee64268fe7261f1d133b38e459ec585c3fee7a615d4e34fbf" exitCode=0
Jul 29 11:33:34.933524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:34.933218    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerDied","Data":"b8df54164b91c98ee64268fe7261f1d133b38e459ec585c3fee7a615d4e34fbf"}
Jul 29 11:33:34.933524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:34.933277    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr" event={"ID":"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef","Type":"ContainerDied","Data":"b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc"}
Jul 29 11:33:34.933524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:34.933319    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6426251a76c00b73be194c53332766d89f7e83b581d138c1428e7a86ddf6abc"
Jul 29 11:33:34.943218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:34.943194    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:33:35.078278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078224    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config\") pod \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") "
Jul 29 11:33:35.078278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078256    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location\") pod \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") "
Jul 29 11:33:35.078448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078312    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") pod \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") "
Jul 29 11:33:35.078448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078347    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert\") pod \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") "
Jul 29 11:33:35.078448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078374    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf9r8\" (UniqueName: \"kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8\") pod \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\" (UID: \"baf4df9f-9a99-4d0e-ac1c-88310aa6fbef\") "
Jul 29 11:33:35.078668 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078646    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:33:35.078726 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078591    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:33:35.078726 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.078699    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:33:35.080295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.080270    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:33:35.080417 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.080389    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8" (OuterVolumeSpecName: "kube-api-access-qf9r8") pod "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" (UID: "baf4df9f-9a99-4d0e-ac1c-88310aa6fbef"). InnerVolumeSpecName "kube-api-access-qf9r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:33:35.179051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.179030    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9r8\" (UniqueName: \"kubernetes.io/projected/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-api-access-qf9r8\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:33:35.179051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.179050    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:33:35.179201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.179059    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:33:35.179201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.179068    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:33:35.179201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.179078    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:33:35.670640 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670612    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:33:35.670952 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670939    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="storage-initializer"
Jul 29 11:33:35.671009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670953    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="storage-initializer"
Jul 29 11:33:35.671009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670968    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container"
Jul 29 11:33:35.671009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670974    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container"
Jul 29 11:33:35.671009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670983    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kube-rbac-proxy"
Jul 29 11:33:35.671009 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.670989    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kube-rbac-proxy"
Jul 29 11:33:35.671214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.671042    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kube-rbac-proxy"
Jul 29 11:33:35.671214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.671051    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" containerName="kserve-container"
Jul 29 11:33:35.674370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.674354    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.676826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.676808    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 29 11:33:35.676826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.676816    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 29 11:33:35.691005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.690984    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:33:35.782801 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.782777    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.782897 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.782832    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.782940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.782916    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwrz\" (UniqueName: \"kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.782976 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.782950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.782976 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.782971    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884211 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884311 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884225    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwrz\" (UniqueName: \"kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884311 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884247    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884419 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884419 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884414    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884794 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884770    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.884999 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.884977    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.885060 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.885019    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.886474 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.886455    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.891924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.891905    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwrz\" (UniqueName: \"kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:35.937322 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.937271    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"
Jul 29 11:33:35.958970 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.958950    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:33:35.962112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.962093    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-4tztr"]
Jul 29 11:33:35.984251 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:35.984230    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:36.111668 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:36.111644    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:33:36.113737 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:33:36.113714    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13a4acd_6975_4d9a_b354_69c62c5e26fc.slice/crio-e51ae5ef8f2cb043d1be99f73b2c22e4931431098a58a3eca2eca1b1bd5918b1 WatchSource:0}: Error finding container e51ae5ef8f2cb043d1be99f73b2c22e4931431098a58a3eca2eca1b1bd5918b1: Status 404 returned error can't find the container with id e51ae5ef8f2cb043d1be99f73b2c22e4931431098a58a3eca2eca1b1bd5918b1
Jul 29 11:33:36.728849 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:36.728816    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf4df9f-9a99-4d0e-ac1c-88310aa6fbef" path="/var/lib/kubelet/pods/baf4df9f-9a99-4d0e-ac1c-88310aa6fbef/volumes"
Jul 29 11:33:36.943092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:36.943057    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerStarted","Data":"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"}
Jul 29 11:33:36.943493 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:36.943146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerStarted","Data":"e51ae5ef8f2cb043d1be99f73b2c22e4931431098a58a3eca2eca1b1bd5918b1"}
Jul 29 11:33:39.957876 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:39.957845    2551 generic.go:356] "Generic (PLEG): container finished" podID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerID="2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa" exitCode=0
Jul 29 11:33:39.958195 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:39.957916    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerDied","Data":"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"}
Jul 29 11:33:40.963561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:40.963530    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerStarted","Data":"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"}
Jul 29 11:33:40.963981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:40.963588    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerStarted","Data":"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"}
Jul 29 11:33:40.963981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:40.963783    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:40.983281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:40.983238    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podStartSLOduration=2.284759441 podStartE2EDuration="5.98322413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698464689s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:33:40.981786661 +0000 UTC m=+3872.884015082" watchObservedRunningTime="2026-07-29 11:33:40.98322413 +0000 UTC m=+3872.885452527"
Jul 29 11:33:41.968075 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:41.968046    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:33:47.977221 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:33:47.977184    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:34:08.900039 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:08.900011    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:34:08.909640 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:08.909620    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:34:17.978986 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:17.978939    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 29 11:34:27.977896 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:27.977858    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 29 11:34:37.977785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:37.977749    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 29 11:34:47.978680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:47.978642    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 29 11:34:58.081122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:34:58.081081    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:35:05.788796 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:05.788714    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:35:05.789239 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:05.788986    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" containerID="cri-o://87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761" gracePeriod=30
Jul 29 11:35:05.789239 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:05.789015    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kube-rbac-proxy" containerID="cri-o://99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b" gracePeriod=30
Jul 29 11:35:06.349217 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:06.349185    2551 generic.go:356] "Generic (PLEG): container finished" podID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerID="99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b" exitCode=2
Jul 29 11:35:06.349370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:06.349258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerDied","Data":"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"}
Jul 29 11:35:07.973210 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:07.973169    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 29 11:35:07.978312 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:07.978282    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 29 11:35:09.640190 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.640169    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:35:09.699941 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.699912    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config\") pod \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") "
Jul 29 11:35:09.700083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.699964    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zwrz\" (UniqueName: \"kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz\") pod \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") "
Jul 29 11:35:09.700083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.699992    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls\") pod \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") "
Jul 29 11:35:09.700083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.700037    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert\") pod \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") "
Jul 29 11:35:09.700286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.700104    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location\") pod \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\" (UID: \"c13a4acd-6975-4d9a-b354-69c62c5e26fc\") "
Jul 29 11:35:09.700429 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.700395    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c13a4acd-6975-4d9a-b354-69c62c5e26fc" (UID: "c13a4acd-6975-4d9a-b354-69c62c5e26fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:35:09.700515 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.700475    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c13a4acd-6975-4d9a-b354-69c62c5e26fc" (UID: "c13a4acd-6975-4d9a-b354-69c62c5e26fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:35:09.700515 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.700498    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c13a4acd-6975-4d9a-b354-69c62c5e26fc" (UID: "c13a4acd-6975-4d9a-b354-69c62c5e26fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:35:09.701957 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.701937    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz" (OuterVolumeSpecName: "kube-api-access-7zwrz") pod "c13a4acd-6975-4d9a-b354-69c62c5e26fc" (UID: "c13a4acd-6975-4d9a-b354-69c62c5e26fc"). InnerVolumeSpecName "kube-api-access-7zwrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:35:09.702072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.702055    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c13a4acd-6975-4d9a-b354-69c62c5e26fc" (UID: "c13a4acd-6975-4d9a-b354-69c62c5e26fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:35:09.801055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.800993    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:35:09.801055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.801017    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwrz\" (UniqueName: \"kubernetes.io/projected/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kube-api-access-7zwrz\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:35:09.801055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.801031    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c13a4acd-6975-4d9a-b354-69c62c5e26fc-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:35:09.801055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.801046    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c13a4acd-6975-4d9a-b354-69c62c5e26fc-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:35:09.801281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:09.801059    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c13a4acd-6975-4d9a-b354-69c62c5e26fc-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:35:10.368955 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.368922    2551 generic.go:356] "Generic (PLEG): container finished" podID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerID="87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761" exitCode=0
Jul 29 11:35:10.369155 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.368976    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerDied","Data":"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"}
Jul 29 11:35:10.369155 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.369005    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"
Jul 29 11:35:10.369155 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.369026    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz" event={"ID":"c13a4acd-6975-4d9a-b354-69c62c5e26fc","Type":"ContainerDied","Data":"e51ae5ef8f2cb043d1be99f73b2c22e4931431098a58a3eca2eca1b1bd5918b1"}
Jul 29 11:35:10.369155 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.369053    2551 scope.go:117] "RemoveContainer" containerID="99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"
Jul 29 11:35:10.378622 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.378605    2551 scope.go:117] "RemoveContainer" containerID="87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"
Jul 29 11:35:10.387184 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.387165    2551 scope.go:117] "RemoveContainer" containerID="2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"
Jul 29 11:35:10.391156 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.391112    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:35:10.395698 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.395665    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jxjcz"]
Jul 29 11:35:10.396463 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.396444    2551 scope.go:117] "RemoveContainer" containerID="99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"
Jul 29 11:35:10.396775 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:35:10.396750    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b\": container with ID starting with 99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b not found: ID does not exist" containerID="99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"
Jul 29 11:35:10.396818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.396783    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b"} err="failed to get container status \"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b\": rpc error: code = NotFound desc = could not find container \"99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b\": container with ID starting with 99971cab15709060b184b979fff2338b4ad13626802a952146e85b30fedf401b not found: ID does not exist"
Jul 29 11:35:10.396818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.396802    2551 scope.go:117] "RemoveContainer" containerID="87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"
Jul 29 11:35:10.397088 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:35:10.397068    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761\": container with ID starting with 87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761 not found: ID does not exist" containerID="87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"
Jul 29 11:35:10.397153 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.397094    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761"} err="failed to get container status \"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761\": rpc error: code = NotFound desc = could not find container \"87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761\": container with ID starting with 87af325859ddce4e0df7d4d5121b0cd42b66f5192ccf5754e4962661d9c04761 not found: ID does not exist"
Jul 29 11:35:10.397153 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.397112    2551 scope.go:117] "RemoveContainer" containerID="2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"
Jul 29 11:35:10.397387 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:35:10.397367    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa\": container with ID starting with 2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa not found: ID does not exist" containerID="2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"
Jul 29 11:35:10.397449 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.397396    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa"} err="failed to get container status \"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa\": rpc error: code = NotFound desc = could not find container \"2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa\": container with ID starting with 2912f58d6653453d247fb24fa50e1bcfe1701c33384709a27aaa5106a83a08aa not found: ID does not exist"
Jul 29 11:35:10.728446 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:10.728404    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" path="/var/lib/kubelet/pods/c13a4acd-6975-4d9a-b354-69c62c5e26fc/volumes"
Jul 29 11:35:18.142260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142226    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142537    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142547    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142558    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="storage-initializer"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142566    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="storage-initializer"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142582    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kube-rbac-proxy"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142588    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kube-rbac-proxy"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142645    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kserve-container"
Jul 29 11:35:18.142652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.142654    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13a4acd-6975-4d9a-b354-69c62c5e26fc" containerName="kube-rbac-proxy"
Jul 29 11:35:18.147384 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.147366    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.149826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.149801    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 29 11:35:18.149961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.149846    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:35:18.149961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.149862    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:35:18.149961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.149875    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:35:18.149961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.149881    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:35:18.150911 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.150889    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:35:18.150995 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.150895    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 29 11:35:18.155991 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.155968    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:35:18.263742 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.263718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n6p\" (UniqueName: \"kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.263869 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.263750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.263869 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.263773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.263990 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.263867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.263990 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.263895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364331 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364311    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n6p\" (UniqueName: \"kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364430 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364430 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364430 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364594 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:35:18.364465    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Jul 29 11:35:18.364594 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364501    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.364594 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:35:18.364521    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls podName:1eef27d1-3b15-41c3-b23c-92cea43a1843 nodeName:}" failed. No retries permitted until 2026-07-29 11:35:18.864502708 +0000 UTC m=+3970.766731084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls") pod "isvc-sklearn-predictor-5476666dbc-fck9w" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843") : secret "isvc-sklearn-predictor-serving-cert" not found
Jul 29 11:35:18.364735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.364718    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.365100 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.365079    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.365272 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.365256    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.372339 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.372321    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n6p\" (UniqueName: \"kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.867607 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.867572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:18.870099 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:18.870065    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") pod \"isvc-sklearn-predictor-5476666dbc-fck9w\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:19.057828 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:19.057798    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:19.189512 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:19.189486    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:35:19.191913 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:35:19.191880    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eef27d1_3b15_41c3_b23c_92cea43a1843.slice/crio-97f15f1a93094495a7c34ea93636c0b7c03821bd73e257b9a68812cffbcf031b WatchSource:0}: Error finding container 97f15f1a93094495a7c34ea93636c0b7c03821bd73e257b9a68812cffbcf031b: Status 404 returned error can't find the container with id 97f15f1a93094495a7c34ea93636c0b7c03821bd73e257b9a68812cffbcf031b
Jul 29 11:35:19.414666 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:19.414588    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerStarted","Data":"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"}
Jul 29 11:35:19.414786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:19.414702    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerStarted","Data":"97f15f1a93094495a7c34ea93636c0b7c03821bd73e257b9a68812cffbcf031b"}
Jul 29 11:35:23.434878 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:23.434843    2551 generic.go:356] "Generic (PLEG): container finished" podID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerID="7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918" exitCode=0
Jul 29 11:35:23.435275 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:23.434930    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerDied","Data":"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"}
Jul 29 11:35:24.441385 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:24.441348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerStarted","Data":"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"}
Jul 29 11:35:24.441780 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:24.441411    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerStarted","Data":"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"}
Jul 29 11:35:24.441780 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:24.441653    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:24.460169 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:24.460099    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podStartSLOduration=2.774885338 podStartE2EDuration="6.460084556s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685199218s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:35:24.45917667 +0000 UTC m=+3976.361405068" watchObservedRunningTime="2026-07-29 11:35:24.460084556 +0000 UTC m=+3976.362313009"
Jul 29 11:35:25.445728 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:25.445698    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:25.446917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:25.446888    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:35:26.450361 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:26.450322    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:35:31.455235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:31.455206    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:35:31.455715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:31.455686    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:35:41.456104 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:41.456065    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:35:51.456456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:35:51.456414    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:01.455958 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:01.455920    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:11.455998 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:11.455961    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:21.456427 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:21.456388    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:31.455922 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:31.455829    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:41.456321 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:41.456290    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:36:48.259687 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:48.259655    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:36:48.260088 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:48.259951    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" containerID="cri-o://dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee" gracePeriod=30
Jul 29 11:36:48.260088 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:48.259959    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kube-rbac-proxy" containerID="cri-o://d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7" gracePeriod=30
Jul 29 11:36:48.797489 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:48.797460    2551 generic.go:356] "Generic (PLEG): container finished" podID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerID="d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7" exitCode=2
Jul 29 11:36:48.797638 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:48.797524    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerDied","Data":"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"}
Jul 29 11:36:51.451501 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:51.451461    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Jul 29 11:36:51.455789 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:51.455767    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 29 11:36:52.308013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.307988    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:36:52.362748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.362678    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location\") pod \"1eef27d1-3b15-41c3-b23c-92cea43a1843\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") "
Jul 29 11:36:52.362748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.362726    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") pod \"1eef27d1-3b15-41c3-b23c-92cea43a1843\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") "
Jul 29 11:36:52.362937 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.362760    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config\") pod \"1eef27d1-3b15-41c3-b23c-92cea43a1843\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") "
Jul 29 11:36:52.362937 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.362778    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert\") pod \"1eef27d1-3b15-41c3-b23c-92cea43a1843\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") "
Jul 29 11:36:52.362937 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.362842    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2n6p\" (UniqueName: \"kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p\") pod \"1eef27d1-3b15-41c3-b23c-92cea43a1843\" (UID: \"1eef27d1-3b15-41c3-b23c-92cea43a1843\") "
Jul 29 11:36:52.363067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.363040    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1eef27d1-3b15-41c3-b23c-92cea43a1843" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:36:52.363192 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.363168    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1eef27d1-3b15-41c3-b23c-92cea43a1843" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:36:52.363339 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.363185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1eef27d1-3b15-41c3-b23c-92cea43a1843" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:36:52.364786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.364768    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p" (OuterVolumeSpecName: "kube-api-access-t2n6p") pod "1eef27d1-3b15-41c3-b23c-92cea43a1843" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843"). InnerVolumeSpecName "kube-api-access-t2n6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:36:52.364936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.364918    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1eef27d1-3b15-41c3-b23c-92cea43a1843" (UID: "1eef27d1-3b15-41c3-b23c-92cea43a1843"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:36:52.464303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.464265    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eef27d1-3b15-41c3-b23c-92cea43a1843-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:36:52.464303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.464294    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:36:52.464303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.464304    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1eef27d1-3b15-41c3-b23c-92cea43a1843-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:36:52.464727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.464315    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2n6p\" (UniqueName: \"kubernetes.io/projected/1eef27d1-3b15-41c3-b23c-92cea43a1843-kube-api-access-t2n6p\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:36:52.464727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.464324    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eef27d1-3b15-41c3-b23c-92cea43a1843-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:36:52.817139 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.817101    2551 generic.go:356] "Generic (PLEG): container finished" podID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerID="dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee" exitCode=0
Jul 29 11:36:52.817328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.817189    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerDied","Data":"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"}
Jul 29 11:36:52.817328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.817202    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"
Jul 29 11:36:52.817328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.817295    2551 scope.go:117] "RemoveContainer" containerID="d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"
Jul 29 11:36:52.817506 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.817283    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w" event={"ID":"1eef27d1-3b15-41c3-b23c-92cea43a1843","Type":"ContainerDied","Data":"97f15f1a93094495a7c34ea93636c0b7c03821bd73e257b9a68812cffbcf031b"}
Jul 29 11:36:52.825573 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.825557    2551 scope.go:117] "RemoveContainer" containerID="dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"
Jul 29 11:36:52.832713 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.832676    2551 scope.go:117] "RemoveContainer" containerID="7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"
Jul 29 11:36:52.835396 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.835378    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:36:52.839905 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.839881    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5476666dbc-fck9w"]
Jul 29 11:36:52.843444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.843427    2551 scope.go:117] "RemoveContainer" containerID="d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"
Jul 29 11:36:52.843919 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:36:52.843897    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7\": container with ID starting with d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7 not found: ID does not exist" containerID="d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"
Jul 29 11:36:52.843987 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.843929    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7"} err="failed to get container status \"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7\": rpc error: code = NotFound desc = could not find container \"d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7\": container with ID starting with d64db053a84ca8160d8416f37bc1895b6b5a40ce2b4ff02c29a8b2c5d84695e7 not found: ID does not exist"
Jul 29 11:36:52.843987 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.843953    2551 scope.go:117] "RemoveContainer" containerID="dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"
Jul 29 11:36:52.844246 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:36:52.844224    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee\": container with ID starting with dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee not found: ID does not exist" containerID="dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"
Jul 29 11:36:52.844310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.844251    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee"} err="failed to get container status \"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee\": rpc error: code = NotFound desc = could not find container \"dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee\": container with ID starting with dd0801776dab15ea5b3ca1c4de30b3091792a9535e8b1c8028e47529526cd1ee not found: ID does not exist"
Jul 29 11:36:52.844310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.844267    2551 scope.go:117] "RemoveContainer" containerID="7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"
Jul 29 11:36:52.844505 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:36:52.844489    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918\": container with ID starting with 7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918 not found: ID does not exist" containerID="7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"
Jul 29 11:36:52.844546 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:52.844509    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918"} err="failed to get container status \"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918\": rpc error: code = NotFound desc = could not find container \"7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918\": container with ID starting with 7e12b7984f6ca1dc18a545a9166bede8ece88ac055e4dee869684b3c47818918 not found: ID does not exist"
Jul 29 11:36:53.418529 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418501    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:36:53.418817 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418804    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kube-rbac-proxy"
Jul 29 11:36:53.418858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418818    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kube-rbac-proxy"
Jul 29 11:36:53.418858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418839    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="storage-initializer"
Jul 29 11:36:53.418858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418844    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="storage-initializer"
Jul 29 11:36:53.418858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418852    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container"
Jul 29 11:36:53.418858 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418858    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container"
Jul 29 11:36:53.419014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418920    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kube-rbac-proxy"
Jul 29 11:36:53.419014 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.418931    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" containerName="kserve-container"
Jul 29 11:36:53.423295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.423277    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.425764 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.425745    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:36:53.425885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.425749    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:36:53.425885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.425749    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:36:53.426845 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.426827    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 29 11:36:53.426936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.426851    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 29 11:36:53.426936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.426866    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:36:53.427033 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.426965    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:36:53.431941 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.431917    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:36:53.475035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.475007    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.475351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.475059    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.475351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.475090    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.475351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.475145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6x9w\" (UniqueName: \"kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.475351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.475225    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.575914 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.575881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x9w\" (UniqueName: \"kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.575943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.575961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.575992    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.576009    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576387 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.576367    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576659 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.576639    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.576754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.576676    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.578509 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.578487    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.583808 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.583785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x9w\" (UniqueName: \"kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.734694 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.734672    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:53.864900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:53.864874    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:36:53.866596 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:36:53.866561    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2b3ef_9f05_4bc5_b068_925a6996da77.slice/crio-88c2620940fcdf69f81eeb019622ff6ef5a81b4375851989bb6a514639993d09 WatchSource:0}: Error finding container 88c2620940fcdf69f81eeb019622ff6ef5a81b4375851989bb6a514639993d09: Status 404 returned error can't find the container with id 88c2620940fcdf69f81eeb019622ff6ef5a81b4375851989bb6a514639993d09
Jul 29 11:36:54.728175 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:54.728143    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eef27d1-3b15-41c3-b23c-92cea43a1843" path="/var/lib/kubelet/pods/1eef27d1-3b15-41c3-b23c-92cea43a1843/volumes"
Jul 29 11:36:54.829230 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:54.829192    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerStarted","Data":"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"}
Jul 29 11:36:54.829407 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:54.829260    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerStarted","Data":"88c2620940fcdf69f81eeb019622ff6ef5a81b4375851989bb6a514639993d09"}
Jul 29 11:36:57.844536 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:57.844461    2551 generic.go:356] "Generic (PLEG): container finished" podID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerID="460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee" exitCode=0
Jul 29 11:36:57.844909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:57.844528    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerDied","Data":"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"}
Jul 29 11:36:58.851044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:58.851008    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerStarted","Data":"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"}
Jul 29 11:36:58.851442 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:58.851074    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerStarted","Data":"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"}
Jul 29 11:36:58.851442 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:58.851283    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:36:58.870658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:58.870612    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" podStartSLOduration=2.304376378 podStartE2EDuration="5.870598702s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.566222324s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:36:58.870159285 +0000 UTC m=+4070.772387682" watchObservedRunningTime="2026-07-29 11:36:58.870598702 +0000 UTC m=+4070.772827099"
Jul 29 11:36:59.855781 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:36:59.855753    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:37:05.865906 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:05.865878    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:37:35.882145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:35.882097    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 29 11:37:45.869774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:45.869737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:37:53.483920 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:53.483887    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:37:53.484437 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:53.484191    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kserve-container" containerID="cri-o://be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7" gracePeriod=30
Jul 29 11:37:53.484437 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:53.484198    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kube-rbac-proxy" containerID="cri-o://b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a" gracePeriod=30
Jul 29 11:37:54.086141 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:54.086094    2551 generic.go:356] "Generic (PLEG): container finished" podID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerID="b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a" exitCode=2
Jul 29 11:37:54.086320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:54.086176    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerDied","Data":"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"}
Jul 29 11:37:55.860696 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:55.860658    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Jul 29 11:37:58.669399 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.669365    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:37:58.672450 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.672435    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.674843 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.674821    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 29 11:37:58.674955 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.674828    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:37:58.681740 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.681705    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:37:58.767649 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.767626    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.767757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.767658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.767757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.767689    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.767757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.767746    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.767878 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.767785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8f9\" (UniqueName: \"kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868145 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868168    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8f9\" (UniqueName: \"kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868200    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868220    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868663 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868639    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868908 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.868908 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.868900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.870637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.870611    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.876346 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.876321    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8f9\" (UniqueName: \"kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9\") pod \"isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:58.982554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:58.982522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:37:59.110935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.110910    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:37:59.112825 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:37:59.112794    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f5cb6a_66c1_4e9b_ab05_e316c293c5cd.slice/crio-2c6f7e3fe6e1d15f4a9e069bd9d60f29de828794af74bd8b63bda3a877e7bc51 WatchSource:0}: Error finding container 2c6f7e3fe6e1d15f4a9e069bd9d60f29de828794af74bd8b63bda3a877e7bc51: Status 404 returned error can't find the container with id 2c6f7e3fe6e1d15f4a9e069bd9d60f29de828794af74bd8b63bda3a877e7bc51
Jul 29 11:37:59.427455 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.427431    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:37:59.473972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.473942    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6x9w\" (UniqueName: \"kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w\") pod \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") "
Jul 29 11:37:59.474115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.473982    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert\") pod \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") "
Jul 29 11:37:59.474115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474032    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls\") pod \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") "
Jul 29 11:37:59.474115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474056    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location\") pod \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") "
Jul 29 11:37:59.474115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474072    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config\") pod \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\" (UID: \"f8c2b3ef-9f05-4bc5-b068-925a6996da77\") "
Jul 29 11:37:59.474461 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474427    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8c2b3ef-9f05-4bc5-b068-925a6996da77" (UID: "f8c2b3ef-9f05-4bc5-b068-925a6996da77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:37:59.474590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8c2b3ef-9f05-4bc5-b068-925a6996da77" (UID: "f8c2b3ef-9f05-4bc5-b068-925a6996da77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:37:59.474590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.474550    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8c2b3ef-9f05-4bc5-b068-925a6996da77" (UID: "f8c2b3ef-9f05-4bc5-b068-925a6996da77"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:37:59.476064 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.476042    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w" (OuterVolumeSpecName: "kube-api-access-m6x9w") pod "f8c2b3ef-9f05-4bc5-b068-925a6996da77" (UID: "f8c2b3ef-9f05-4bc5-b068-925a6996da77"). InnerVolumeSpecName "kube-api-access-m6x9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:37:59.476064 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.476049    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8c2b3ef-9f05-4bc5-b068-925a6996da77" (UID: "f8c2b3ef-9f05-4bc5-b068-925a6996da77"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:37:59.574825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.574772    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8c2b3ef-9f05-4bc5-b068-925a6996da77-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:37:59.574825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.574793    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:37:59.574825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.574803    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:37:59.574825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.574812    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6x9w\" (UniqueName: \"kubernetes.io/projected/f8c2b3ef-9f05-4bc5-b068-925a6996da77-kube-api-access-m6x9w\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:37:59.574825 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:37:59.574820    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2b3ef-9f05-4bc5-b068-925a6996da77-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:00.116582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.116543    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerStarted","Data":"9cc7edf18a393d14a09e358b42db1592b9c2933a0ab7098d60767713c61d7007"}
Jul 29 11:38:00.117044 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.116617    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerStarted","Data":"2c6f7e3fe6e1d15f4a9e069bd9d60f29de828794af74bd8b63bda3a877e7bc51"}
Jul 29 11:38:00.118584 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.118557    2551 generic.go:356] "Generic (PLEG): container finished" podID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerID="be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7" exitCode=0
Jul 29 11:38:00.118717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.118625    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"
Jul 29 11:38:00.118717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.118637    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerDied","Data":"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"}
Jul 29 11:38:00.118717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.118686    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q" event={"ID":"f8c2b3ef-9f05-4bc5-b068-925a6996da77","Type":"ContainerDied","Data":"88c2620940fcdf69f81eeb019622ff6ef5a81b4375851989bb6a514639993d09"}
Jul 29 11:38:00.118717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.118711    2551 scope.go:117] "RemoveContainer" containerID="b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"
Jul 29 11:38:00.127812 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.127794    2551 scope.go:117] "RemoveContainer" containerID="be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"
Jul 29 11:38:00.135576 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.135557    2551 scope.go:117] "RemoveContainer" containerID="460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"
Jul 29 11:38:00.143059 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.143035    2551 scope.go:117] "RemoveContainer" containerID="b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"
Jul 29 11:38:00.143474 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:38:00.143447    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a\": container with ID starting with b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a not found: ID does not exist" containerID="b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"
Jul 29 11:38:00.143554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.143483    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a"} err="failed to get container status \"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a\": rpc error: code = NotFound desc = could not find container \"b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a\": container with ID starting with b19f81b13ce6272fadd3409172996d10733cfb60c1bbba59795470c775d7f20a not found: ID does not exist"
Jul 29 11:38:00.143554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.143500    2551 scope.go:117] "RemoveContainer" containerID="be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"
Jul 29 11:38:00.143781 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:38:00.143760    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7\": container with ID starting with be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7 not found: ID does not exist" containerID="be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"
Jul 29 11:38:00.143832 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.143787    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7"} err="failed to get container status \"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7\": rpc error: code = NotFound desc = could not find container \"be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7\": container with ID starting with be5a19d7049b4045e8916314f293847ef281b0d826b0a5d71150bcd8c1c48aa7 not found: ID does not exist"
Jul 29 11:38:00.143832 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.143805    2551 scope.go:117] "RemoveContainer" containerID="460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"
Jul 29 11:38:00.144058 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:38:00.144041    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee\": container with ID starting with 460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee not found: ID does not exist" containerID="460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"
Jul 29 11:38:00.144110 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.144065    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee"} err="failed to get container status \"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee\": rpc error: code = NotFound desc = could not find container \"460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee\": container with ID starting with 460c315c2520de54812ba24f3ef09b902e9e50ef76a3d71fa08af07e479a21ee not found: ID does not exist"
Jul 29 11:38:00.148665 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.148644    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:38:00.153788 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.153768    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-97x5q"]
Jul 29 11:38:00.728404 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:00.728376    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" path="/var/lib/kubelet/pods/f8c2b3ef-9f05-4bc5-b068-925a6996da77/volumes"
Jul 29 11:38:05.150924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:05.150852    2551 generic.go:356] "Generic (PLEG): container finished" podID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerID="9cc7edf18a393d14a09e358b42db1592b9c2933a0ab7098d60767713c61d7007" exitCode=0
Jul 29 11:38:05.150924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:05.150896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerDied","Data":"9cc7edf18a393d14a09e358b42db1592b9c2933a0ab7098d60767713c61d7007"}
Jul 29 11:38:06.158117 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:06.158086    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerStarted","Data":"6d4ac5a5cf354794bf3de3d5f2f69e2368ffa2eaca05803d82d7fecbc1aec7a8"}
Jul 29 11:38:06.158517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:06.158155    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerStarted","Data":"7ea8cc97a7b1e3d9cbf143d6029d3a4cb1f80d489a3f15d136e93f626c36c096"}
Jul 29 11:38:06.158517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:06.158363    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:06.178928 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:06.178878    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podStartSLOduration=2.468099484 podStartE2EDuration="8.17886349s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.710764006s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:38:06.176502398 +0000 UTC m=+4138.078730805" watchObservedRunningTime="2026-07-29 11:38:06.17886349 +0000 UTC m=+4138.081091886"
Jul 29 11:38:07.163279 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:07.163247    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:07.164253 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:07.164228    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 29 11:38:08.167964 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:08.167924    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 29 11:38:13.172477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:13.172446    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:13.172956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:13.172930    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 29 11:38:22.490351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:22.490320    2551 scope.go:117] "RemoveContainer" containerID="6241e0b2d1b541124de6f56abf9aa0c307bb4789bd1a0bd151ff0bfe0abccb57"
Jul 29 11:38:22.500619 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:22.500596    2551 scope.go:117] "RemoveContainer" containerID="08ae57a2472dbdc090b02010c96238224d3bb6e97417c9de93a32302a92c191a"
Jul 29 11:38:22.508077 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:22.508058    2551 scope.go:117] "RemoveContainer" containerID="b8df54164b91c98ee64268fe7261f1d133b38e459ec585c3fee7a615d4e34fbf"
Jul 29 11:38:23.173303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:23.173274    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:30.326963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:30.326932    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc_10f5cb6a-66c1-4e9b-ab05-e316c293c5cd/kserve-container/0.log"
Jul 29 11:38:30.484343 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:30.484305    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:38:30.484630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:30.484599    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container" containerID="cri-o://7ea8cc97a7b1e3d9cbf143d6029d3a4cb1f80d489a3f15d136e93f626c36c096" gracePeriod=30
Jul 29 11:38:30.484729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:30.484679    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kube-rbac-proxy" containerID="cri-o://6d4ac5a5cf354794bf3de3d5f2f69e2368ffa2eaca05803d82d7fecbc1aec7a8" gracePeriod=30
Jul 29 11:38:31.279348 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.279316    2551 generic.go:356] "Generic (PLEG): container finished" podID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerID="6d4ac5a5cf354794bf3de3d5f2f69e2368ffa2eaca05803d82d7fecbc1aec7a8" exitCode=2
Jul 29 11:38:31.279348 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.279342    2551 generic.go:356] "Generic (PLEG): container finished" podID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerID="7ea8cc97a7b1e3d9cbf143d6029d3a4cb1f80d489a3f15d136e93f626c36c096" exitCode=0
Jul 29 11:38:31.279547 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.279376    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerDied","Data":"6d4ac5a5cf354794bf3de3d5f2f69e2368ffa2eaca05803d82d7fecbc1aec7a8"}
Jul 29 11:38:31.279547 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.279437    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerDied","Data":"7ea8cc97a7b1e3d9cbf143d6029d3a4cb1f80d489a3f15d136e93f626c36c096"}
Jul 29 11:38:31.331351 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.331328    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:31.345413 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345391    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls\") pod \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") "
Jul 29 11:38:31.345517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345432    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location\") pod \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") "
Jul 29 11:38:31.345517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345451    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8f9\" (UniqueName: \"kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9\") pod \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") "
Jul 29 11:38:31.345517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345497    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config\") pod \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") "
Jul 29 11:38:31.345680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345547    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert\") pod \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\" (UID: \"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd\") "
Jul 29 11:38:31.345896 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345866    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" (UID: "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:38:31.345974 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.345896    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" (UID: "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:38:31.347627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.347606    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" (UID: "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:38:31.347720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.347682    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9" (OuterVolumeSpecName: "kube-api-access-qx8f9") pod "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" (UID: "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd"). InnerVolumeSpecName "kube-api-access-qx8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:38:31.348586 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.348568    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" (UID: "10f5cb6a-66c1-4e9b-ab05-e316c293c5cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:38:31.446240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.446160    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:31.446240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.446186    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:31.446240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.446199    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:31.446240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.446212    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8f9\" (UniqueName: \"kubernetes.io/projected/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-api-access-qx8f9\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:31.446240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:31.446224    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:38:32.285445 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.285417    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"
Jul 29 11:38:32.285611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.285409    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc" event={"ID":"10f5cb6a-66c1-4e9b-ab05-e316c293c5cd","Type":"ContainerDied","Data":"2c6f7e3fe6e1d15f4a9e069bd9d60f29de828794af74bd8b63bda3a877e7bc51"}
Jul 29 11:38:32.285611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.285560    2551 scope.go:117] "RemoveContainer" containerID="6d4ac5a5cf354794bf3de3d5f2f69e2368ffa2eaca05803d82d7fecbc1aec7a8"
Jul 29 11:38:32.298993 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.298974    2551 scope.go:117] "RemoveContainer" containerID="7ea8cc97a7b1e3d9cbf143d6029d3a4cb1f80d489a3f15d136e93f626c36c096"
Jul 29 11:38:32.306610 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.306591    2551 scope.go:117] "RemoveContainer" containerID="9cc7edf18a393d14a09e358b42db1592b9c2933a0ab7098d60767713c61d7007"
Jul 29 11:38:32.312268 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.312117    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:38:32.315856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.315836    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-568d8b5f95-zz9fc"]
Jul 29 11:38:32.727721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:32.727688    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" path="/var/lib/kubelet/pods/10f5cb6a-66c1-4e9b-ab05-e316c293c5cd/volumes"
Jul 29 11:38:35.622752 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.622717    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623022    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623032    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623048    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kube-rbac-proxy"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623053    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kube-rbac-proxy"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623062    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kube-rbac-proxy"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623068    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kube-rbac-proxy"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623082    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="storage-initializer"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623086    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="storage-initializer"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623093    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kserve-container"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623098    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kserve-container"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623105    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="storage-initializer"
Jul 29 11:38:35.623118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623110    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="storage-initializer"
Jul 29 11:38:35.623520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623191    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kserve-container"
Jul 29 11:38:35.623520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623200    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kserve-container"
Jul 29 11:38:35.623520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623207    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2b3ef-9f05-4bc5-b068-925a6996da77" containerName="kube-rbac-proxy"
Jul 29 11:38:35.623520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.623213    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f5cb6a-66c1-4e9b-ab05-e316c293c5cd" containerName="kube-rbac-proxy"
Jul 29 11:38:35.627838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.627822    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.630119 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.630088    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 29 11:38:35.630329 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.630310    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:38:35.630421 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.630401    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:38:35.630499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.630485    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:38:35.630591 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.630576    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:38:35.631173 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.631160    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:38:35.631216 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.631182    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:38:35.636460 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.636436    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:38:35.677465 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.677436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw68n\" (UniqueName: \"kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.677590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.677470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.677590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.677498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.677590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.677540    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.677590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.677565    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.778925 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.778897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.778938    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw68n\" (UniqueName: \"kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.778962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.778990    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.779028    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779496 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.779472    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.779626    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.779702 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.779687    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.781468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.781436    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.786906 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.786879    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw68n\" (UniqueName: \"kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:35.938182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:35.938089    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:36.070174 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:36.070146    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:38:36.072318 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:38:36.072290    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6e727c_8715_42f7_b18c_312f89c6b6e5.slice/crio-3270aa3c8f03a38453ac2da3f57c8575f084c4859f674d10c7805cbc8a433a73 WatchSource:0}: Error finding container 3270aa3c8f03a38453ac2da3f57c8575f084c4859f674d10c7805cbc8a433a73: Status 404 returned error can't find the container with id 3270aa3c8f03a38453ac2da3f57c8575f084c4859f674d10c7805cbc8a433a73
Jul 29 11:38:36.304274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:36.304243    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerStarted","Data":"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"}
Jul 29 11:38:36.304488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:36.304301    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerStarted","Data":"3270aa3c8f03a38453ac2da3f57c8575f084c4859f674d10c7805cbc8a433a73"}
Jul 29 11:38:40.324543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:40.324458    2551 generic.go:356] "Generic (PLEG): container finished" podID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerID="bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097" exitCode=0
Jul 29 11:38:40.324543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:40.324517    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerDied","Data":"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"}
Jul 29 11:38:41.330812 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:41.330775    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerStarted","Data":"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"}
Jul 29 11:38:41.331397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:41.330840    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerStarted","Data":"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"}
Jul 29 11:38:41.331397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:41.331038    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:41.331397 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:41.331087    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:38:41.351709 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:41.351659    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podStartSLOduration=2.525341681 podStartE2EDuration="6.351645107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826303426s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:38:41.351153739 +0000 UTC m=+4173.253382138" watchObservedRunningTime="2026-07-29 11:38:41.351645107 +0000 UTC m=+4173.253873580"
Jul 29 11:38:47.341266 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:38:47.341230    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:39:08.927634 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:08.927607    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:39:08.938160 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:08.938142    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:39:17.381939 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:17.381894    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 29 11:39:27.343924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:27.343892    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:39:35.726761 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:35.726687    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:39:35.727147 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:35.726930    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container" containerID="cri-o://2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5" gracePeriod=30
Jul 29 11:39:35.727147 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:35.726976    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kube-rbac-proxy" containerID="cri-o://670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24" gracePeriod=30
Jul 29 11:39:36.567873 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:36.567833    2551 generic.go:356] "Generic (PLEG): container finished" podID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerID="670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24" exitCode=2
Jul 29 11:39:36.568050 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:36.567881    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerDied","Data":"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"}
Jul 29 11:39:37.336053 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:37.336017    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 29 11:39:38.383402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:38.383338    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 11:39:40.875704 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.875675    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:39:40.879206 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.879190    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:40.881477 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.881454    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 29 11:39:40.881596 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.881577    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 29 11:39:40.890659 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.890636    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:39:40.924673 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.924646    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:40.924790 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.924681    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:40.924790 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.924748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dp4\" (UniqueName: \"kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:40.924903 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.924800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:40.924903 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:40.924840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.025720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.025673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.025910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.025740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.025910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.025764    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.025910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.025797    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dp4\" (UniqueName: \"kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.025910 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.025819    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.026102 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:41.025939    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 29 11:39:41.026102 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:41.026005    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls podName:335a78fa-ffca-4481-9014-6b399b6f9e8c nodeName:}" failed. No retries permitted until 2026-07-29 11:39:41.525985248 +0000 UTC m=+4233.428213628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls") pod "isvc-sklearn-v2-predictor-59c999cb75-28rzs" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 29 11:39:41.026291 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.026269    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.026494 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.026476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.026530 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.026488    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.036336 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.036314    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dp4\" (UniqueName: \"kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.530637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:41.530602    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:41.530812 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:41.530743    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 29 11:39:41.530812 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:41.530800    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls podName:335a78fa-ffca-4481-9014-6b399b6f9e8c nodeName:}" failed. No retries permitted until 2026-07-29 11:39:42.530785403 +0000 UTC m=+4234.433013778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls") pod "isvc-sklearn-v2-predictor-59c999cb75-28rzs" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 29 11:39:42.272140 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.272106    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:39:42.339930 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.339843    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert\") pod \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") "
Jul 29 11:39:42.339930 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.339884    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config\") pod \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") "
Jul 29 11:39:42.340168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.339968    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location\") pod \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") "
Jul 29 11:39:42.340168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340019    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw68n\" (UniqueName: \"kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n\") pod \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") "
Jul 29 11:39:42.340168 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340044    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls\") pod \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\" (UID: \"ff6e727c-8715-42f7-b18c-312f89c6b6e5\") "
Jul 29 11:39:42.340327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340268    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff6e727c-8715-42f7-b18c-312f89c6b6e5" (UID: "ff6e727c-8715-42f7-b18c-312f89c6b6e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:39:42.340327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340276    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff6e727c-8715-42f7-b18c-312f89c6b6e5" (UID: "ff6e727c-8715-42f7-b18c-312f89c6b6e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:39:42.340431 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340330    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff6e727c-8715-42f7-b18c-312f89c6b6e5" (UID: "ff6e727c-8715-42f7-b18c-312f89c6b6e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:42.340491 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340441    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:39:42.340491 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340461    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:39:42.340491 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.340476    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:39:42.342108 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.342085    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n" (OuterVolumeSpecName: "kube-api-access-bw68n") pod "ff6e727c-8715-42f7-b18c-312f89c6b6e5" (UID: "ff6e727c-8715-42f7-b18c-312f89c6b6e5"). InnerVolumeSpecName "kube-api-access-bw68n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:39:42.342185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.342110    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff6e727c-8715-42f7-b18c-312f89c6b6e5" (UID: "ff6e727c-8715-42f7-b18c-312f89c6b6e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:39:42.441917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.441877    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw68n\" (UniqueName: \"kubernetes.io/projected/ff6e727c-8715-42f7-b18c-312f89c6b6e5-kube-api-access-bw68n\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:39:42.441917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.441914    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff6e727c-8715-42f7-b18c-312f89c6b6e5-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:39:42.542520 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.542482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:42.544907 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.544883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59c999cb75-28rzs\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:42.593039 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.592962    2551 generic.go:356] "Generic (PLEG): container finished" podID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerID="2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5" exitCode=0
Jul 29 11:39:42.593039 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.593022    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"
Jul 29 11:39:42.593234 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.593052    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerDied","Data":"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"}
Jul 29 11:39:42.593234 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.593110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4" event={"ID":"ff6e727c-8715-42f7-b18c-312f89c6b6e5","Type":"ContainerDied","Data":"3270aa3c8f03a38453ac2da3f57c8575f084c4859f674d10c7805cbc8a433a73"}
Jul 29 11:39:42.593234 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.593156    2551 scope.go:117] "RemoveContainer" containerID="670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"
Jul 29 11:39:42.602103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.602080    2551 scope.go:117] "RemoveContainer" containerID="2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"
Jul 29 11:39:42.609448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.609433    2551 scope.go:117] "RemoveContainer" containerID="bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"
Jul 29 11:39:42.615992 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.615970    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:39:42.616764 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.616746    2551 scope.go:117] "RemoveContainer" containerID="670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"
Jul 29 11:39:42.617014 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:42.616997    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24\": container with ID starting with 670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24 not found: ID does not exist" containerID="670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"
Jul 29 11:39:42.617065 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.617022    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24"} err="failed to get container status \"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24\": rpc error: code = NotFound desc = could not find container \"670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24\": container with ID starting with 670abd361cf1cf2c7b24d4ea9141b22c37ef3f7f91616bf07c4d0805b9007e24 not found: ID does not exist"
Jul 29 11:39:42.617065 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.617037    2551 scope.go:117] "RemoveContainer" containerID="2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"
Jul 29 11:39:42.617269 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:42.617251    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5\": container with ID starting with 2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5 not found: ID does not exist" containerID="2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"
Jul 29 11:39:42.617370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.617272    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5"} err="failed to get container status \"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5\": rpc error: code = NotFound desc = could not find container \"2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5\": container with ID starting with 2a4ece86f945d3f682002d6678282fd93008fc8350ee76566bde5c0a6dd79fe5 not found: ID does not exist"
Jul 29 11:39:42.617370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.617285    2551 scope.go:117] "RemoveContainer" containerID="bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"
Jul 29 11:39:42.617586 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:39:42.617565    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097\": container with ID starting with bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097 not found: ID does not exist" containerID="bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"
Jul 29 11:39:42.617911 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.617881    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097"} err="failed to get container status \"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097\": rpc error: code = NotFound desc = could not find container \"bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097\": container with ID starting with bce155d03de753613b1fdc6361a92943c7376e8acee0867d7b2b895d9b988097 not found: ID does not exist"
Jul 29 11:39:42.619766 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.619748    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-gz4g4"]
Jul 29 11:39:42.690274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.690249    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:42.732389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.730004    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" path="/var/lib/kubelet/pods/ff6e727c-8715-42f7-b18c-312f89c6b6e5/volumes"
Jul 29 11:39:42.821092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:42.821064    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:39:42.822850 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:39:42.822823    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a78fa_ffca_4481_9014_6b399b6f9e8c.slice/crio-8ba02e559240a3ab3015117f4fdb590b8090405d8fa2b74aa1360b30aed4d09c WatchSource:0}: Error finding container 8ba02e559240a3ab3015117f4fdb590b8090405d8fa2b74aa1360b30aed4d09c: Status 404 returned error can't find the container with id 8ba02e559240a3ab3015117f4fdb590b8090405d8fa2b74aa1360b30aed4d09c
Jul 29 11:39:43.598183 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:43.598146    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerStarted","Data":"5f98cd54ddc1f3994655d6cbb9f089b2978b6a9c316455020c846344ad8ec15e"}
Jul 29 11:39:43.598588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:43.598221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerStarted","Data":"8ba02e559240a3ab3015117f4fdb590b8090405d8fa2b74aa1360b30aed4d09c"}
Jul 29 11:39:46.612539 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:46.612504    2551 generic.go:356] "Generic (PLEG): container finished" podID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerID="5f98cd54ddc1f3994655d6cbb9f089b2978b6a9c316455020c846344ad8ec15e" exitCode=0
Jul 29 11:39:46.612868 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:46.612558    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerDied","Data":"5f98cd54ddc1f3994655d6cbb9f089b2978b6a9c316455020c846344ad8ec15e"}
Jul 29 11:39:47.618422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:47.618386    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerStarted","Data":"74e2af81e3afdfb63b2f60a65e73666e9fc581805ff25d0354d3aea649628cc6"}
Jul 29 11:39:47.618991 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:47.618457    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerStarted","Data":"1e0584ff0adf455690cf95de844d2e875b86d22f6ea744340b51a1fa1f0f4504"}
Jul 29 11:39:47.618991 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:47.618686    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:47.638248 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:47.638193    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podStartSLOduration=3.980335268 podStartE2EDuration="7.638180084s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657844816s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:39:47.636754877 +0000 UTC m=+4239.538983275" watchObservedRunningTime="2026-07-29 11:39:47.638180084 +0000 UTC m=+4239.540408480"
Jul 29 11:39:48.622828 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:48.622794    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:48.624081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:48.624053    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:39:49.633214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:49.633168    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:39:54.637724 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:54.637693    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:39:54.638252 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:39:54.638225    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:04.638623 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:04.638578    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:14.638313 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:14.638269    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:24.638911 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:24.638871    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:34.639112 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:34.639071    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:44.638726 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:44.638686    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:40:54.638296 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:40:54.638255    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:41:04.639427 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:04.639344    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:41:11.103214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:11.103183    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:41:11.103707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:11.103408    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" containerID="cri-o://1e0584ff0adf455690cf95de844d2e875b86d22f6ea744340b51a1fa1f0f4504" gracePeriod=30
Jul 29 11:41:11.103707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:11.103454    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kube-rbac-proxy" containerID="cri-o://74e2af81e3afdfb63b2f60a65e73666e9fc581805ff25d0354d3aea649628cc6" gracePeriod=30
Jul 29 11:41:11.975290 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:11.975253    2551 generic.go:356] "Generic (PLEG): container finished" podID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerID="74e2af81e3afdfb63b2f60a65e73666e9fc581805ff25d0354d3aea649628cc6" exitCode=2
Jul 29 11:41:11.975464 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:11.975328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerDied","Data":"74e2af81e3afdfb63b2f60a65e73666e9fc581805ff25d0354d3aea649628cc6"}
Jul 29 11:41:14.633184 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:14.633121    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Jul 29 11:41:14.638857 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:14.638828    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 29 11:41:14.993322 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:14.993296    2551 generic.go:356] "Generic (PLEG): container finished" podID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerID="1e0584ff0adf455690cf95de844d2e875b86d22f6ea744340b51a1fa1f0f4504" exitCode=0
Jul 29 11:41:14.993448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:14.993368    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerDied","Data":"1e0584ff0adf455690cf95de844d2e875b86d22f6ea744340b51a1fa1f0f4504"}
Jul 29 11:41:15.044371 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.044347    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:41:15.139410 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139359    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert\") pod \"335a78fa-ffca-4481-9014-6b399b6f9e8c\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") "
Jul 29 11:41:15.139410 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139408    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location\") pod \"335a78fa-ffca-4481-9014-6b399b6f9e8c\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") "
Jul 29 11:41:15.139590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139434    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") pod \"335a78fa-ffca-4481-9014-6b399b6f9e8c\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") "
Jul 29 11:41:15.139590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139462    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config\") pod \"335a78fa-ffca-4481-9014-6b399b6f9e8c\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") "
Jul 29 11:41:15.139590 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139493    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4dp4\" (UniqueName: \"kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4\") pod \"335a78fa-ffca-4481-9014-6b399b6f9e8c\" (UID: \"335a78fa-ffca-4481-9014-6b399b6f9e8c\") "
Jul 29 11:41:15.139760 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139722    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "335a78fa-ffca-4481-9014-6b399b6f9e8c" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:41:15.139803 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139760    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "335a78fa-ffca-4481-9014-6b399b6f9e8c" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:41:15.139839 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.139817    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "335a78fa-ffca-4481-9014-6b399b6f9e8c" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:41:15.141534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.141505    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4" (OuterVolumeSpecName: "kube-api-access-x4dp4") pod "335a78fa-ffca-4481-9014-6b399b6f9e8c" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c"). InnerVolumeSpecName "kube-api-access-x4dp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:41:15.141534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.141520    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "335a78fa-ffca-4481-9014-6b399b6f9e8c" (UID: "335a78fa-ffca-4481-9014-6b399b6f9e8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:41:15.240621 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.240598    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dp4\" (UniqueName: \"kubernetes.io/projected/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-api-access-x4dp4\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:41:15.240621 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.240620    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:41:15.240748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.240629    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/335a78fa-ffca-4481-9014-6b399b6f9e8c-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:41:15.240748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.240638    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/335a78fa-ffca-4481-9014-6b399b6f9e8c-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:41:15.240748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:15.240647    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/335a78fa-ffca-4481-9014-6b399b6f9e8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:41:16.000637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.000607    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"
Jul 29 11:41:16.001045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.000600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs" event={"ID":"335a78fa-ffca-4481-9014-6b399b6f9e8c","Type":"ContainerDied","Data":"8ba02e559240a3ab3015117f4fdb590b8090405d8fa2b74aa1360b30aed4d09c"}
Jul 29 11:41:16.001045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.000756    2551 scope.go:117] "RemoveContainer" containerID="74e2af81e3afdfb63b2f60a65e73666e9fc581805ff25d0354d3aea649628cc6"
Jul 29 11:41:16.010385 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.010364    2551 scope.go:117] "RemoveContainer" containerID="1e0584ff0adf455690cf95de844d2e875b86d22f6ea744340b51a1fa1f0f4504"
Jul 29 11:41:16.018749 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.018732    2551 scope.go:117] "RemoveContainer" containerID="5f98cd54ddc1f3994655d6cbb9f089b2978b6a9c316455020c846344ad8ec15e"
Jul 29 11:41:16.023701 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.023670    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:41:16.027838 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.027809    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59c999cb75-28rzs"]
Jul 29 11:41:16.252943 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.252874    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:41:16.253223 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253211    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253225    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253237    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="storage-initializer"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253243    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="storage-initializer"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253250    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253255    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253264 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253265    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="storage-initializer"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253271    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="storage-initializer"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253280    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253286    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253299    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253303    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253352    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kserve-container"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253361    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253368    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kube-rbac-proxy"
Jul 29 11:41:16.253452 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.253377    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6e727c-8715-42f7-b18c-312f89c6b6e5" containerName="kserve-container"
Jul 29 11:41:16.257809 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.257793    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.260499 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260475    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:41:16.260627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260527    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:41:16.260627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260564    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 29 11:41:16.260627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260574    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:41:16.260627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260600    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:41:16.260627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260609    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 29 11:41:16.260922 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.260909    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:41:16.267584 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.267562    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:41:16.349378 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.349354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.349497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.349387    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2xs\" (UniqueName: \"kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.349497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.349410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.349497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.349434    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.349604 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.349502    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.449950 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.449927    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.449982    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450025    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2xs\" (UniqueName: \"kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450061    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450466 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450444    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450727 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450709    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.450771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.450726    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.452267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.452247    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.459643 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.459620    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2xs\" (UniqueName: \"kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs\") pod \"isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.568483 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.568424    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:16.698167 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.698143    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:41:16.700442 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:41:16.700416    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3360e9_fc71_49d5_b91e_832f129b0913.slice/crio-570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff WatchSource:0}: Error finding container 570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff: Status 404 returned error can't find the container with id 570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff
Jul 29 11:41:16.729067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:16.729039    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335a78fa-ffca-4481-9014-6b399b6f9e8c" path="/var/lib/kubelet/pods/335a78fa-ffca-4481-9014-6b399b6f9e8c/volumes"
Jul 29 11:41:17.008522 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:17.008494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerStarted","Data":"95fdfe594f6762b68fc0bb14b7031ba9d5d79d4cfbf84a23e5d29fc26056571e"}
Jul 29 11:41:17.008875 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:17.008544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerStarted","Data":"570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff"}
Jul 29 11:41:21.028438 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:21.028368    2551 generic.go:356] "Generic (PLEG): container finished" podID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerID="95fdfe594f6762b68fc0bb14b7031ba9d5d79d4cfbf84a23e5d29fc26056571e" exitCode=0
Jul 29 11:41:21.028438 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:21.028420    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerDied","Data":"95fdfe594f6762b68fc0bb14b7031ba9d5d79d4cfbf84a23e5d29fc26056571e"}
Jul 29 11:41:22.034907 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.034865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerStarted","Data":"b38d7a22451a4806d0032d10b57a567cebd52dc7b5f3130e586bfd804c804a25"}
Jul 29 11:41:22.035353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.034930    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerStarted","Data":"88c06e5440d8f4d6ef84494e305ee3e9e04bf83e31c6df2e7a9411488dbdf0ca"}
Jul 29 11:41:22.035353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.035224    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:22.035353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.035284    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:22.036524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.036499    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:41:22.056563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:22.056526    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podStartSLOduration=2.047760098 podStartE2EDuration="6.056512443s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.008752345s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:41:22.05468003 +0000 UTC m=+4333.956908439" watchObservedRunningTime="2026-07-29 11:41:22.056512443 +0000 UTC m=+4333.958740905"
Jul 29 11:41:23.038542 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:23.038502    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:41:28.042528 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:28.042501    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:41:28.042968 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:28.042943    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:41:38.043357 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:38.043319    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:41:48.043853 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:48.043811    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:41:58.043146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:41:58.043086    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:42:08.042970 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:08.042930    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:42:18.043295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:18.043256    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:42:28.043747 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:28.043709    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 29 11:42:33.723355 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:33.723276    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:42:36.324919 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:36.324877    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:42:36.325369 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:36.325140    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container" containerID="cri-o://88c06e5440d8f4d6ef84494e305ee3e9e04bf83e31c6df2e7a9411488dbdf0ca" gracePeriod=30
Jul 29 11:42:36.325369 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:36.325177    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kube-rbac-proxy" containerID="cri-o://b38d7a22451a4806d0032d10b57a567cebd52dc7b5f3130e586bfd804c804a25" gracePeriod=30
Jul 29 11:42:37.351851 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:37.351822    2551 generic.go:356] "Generic (PLEG): container finished" podID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerID="b38d7a22451a4806d0032d10b57a567cebd52dc7b5f3130e586bfd804c804a25" exitCode=2
Jul 29 11:42:37.352258 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:37.351904    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerDied","Data":"b38d7a22451a4806d0032d10b57a567cebd52dc7b5f3130e586bfd804c804a25"}
Jul 29 11:42:38.039603 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:38.039563    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 29 11:42:40.367073 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.367041    2551 generic.go:356] "Generic (PLEG): container finished" podID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerID="88c06e5440d8f4d6ef84494e305ee3e9e04bf83e31c6df2e7a9411488dbdf0ca" exitCode=0
Jul 29 11:42:40.367436 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.367113    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerDied","Data":"88c06e5440d8f4d6ef84494e305ee3e9e04bf83e31c6df2e7a9411488dbdf0ca"}
Jul 29 11:42:40.367436 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.367184    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg" event={"ID":"ce3360e9-fc71-49d5-b91e-832f129b0913","Type":"ContainerDied","Data":"570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff"}
Jul 29 11:42:40.367436 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.367213    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570be22a6255f1af2563f1055a2712f8456e0a1d74c3e8438dce2c0f2cef12ff"
Jul 29 11:42:40.370687 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.370668    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:42:40.472337 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv2xs\" (UniqueName: \"kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs\") pod \"ce3360e9-fc71-49d5-b91e-832f129b0913\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") "
Jul 29 11:42:40.472459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472344    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls\") pod \"ce3360e9-fc71-49d5-b91e-832f129b0913\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") "
Jul 29 11:42:40.472459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472427    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert\") pod \"ce3360e9-fc71-49d5-b91e-832f129b0913\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") "
Jul 29 11:42:40.472459 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472445    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location\") pod \"ce3360e9-fc71-49d5-b91e-832f129b0913\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") "
Jul 29 11:42:40.472574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472466    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config\") pod \"ce3360e9-fc71-49d5-b91e-832f129b0913\" (UID: \"ce3360e9-fc71-49d5-b91e-832f129b0913\") "
Jul 29 11:42:40.472802 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472739    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce3360e9-fc71-49d5-b91e-832f129b0913" (UID: "ce3360e9-fc71-49d5-b91e-832f129b0913"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:42:40.472870 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472837    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce3360e9-fc71-49d5-b91e-832f129b0913" (UID: "ce3360e9-fc71-49d5-b91e-832f129b0913"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:42:40.472927 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.472900    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce3360e9-fc71-49d5-b91e-832f129b0913" (UID: "ce3360e9-fc71-49d5-b91e-832f129b0913"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:42:40.474434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.474414    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs" (OuterVolumeSpecName: "kube-api-access-rv2xs") pod "ce3360e9-fc71-49d5-b91e-832f129b0913" (UID: "ce3360e9-fc71-49d5-b91e-832f129b0913"). InnerVolumeSpecName "kube-api-access-rv2xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:42:40.474502 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.474456    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce3360e9-fc71-49d5-b91e-832f129b0913" (UID: "ce3360e9-fc71-49d5-b91e-832f129b0913"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:42:40.573508 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.573488    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:42:40.573611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.573511    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce3360e9-fc71-49d5-b91e-832f129b0913-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:42:40.573611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.573525    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:42:40.573611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.573537    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2xs\" (UniqueName: \"kubernetes.io/projected/ce3360e9-fc71-49d5-b91e-832f129b0913-kube-api-access-rv2xs\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:42:40.573611 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:40.573551    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce3360e9-fc71-49d5-b91e-832f129b0913-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:42:41.371366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:41.371337    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"
Jul 29 11:42:41.393307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:41.391322    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:42:41.397019 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:41.396993    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-64b45f7998-445fg"]
Jul 29 11:42:42.729062 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:42:42.729030    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" path="/var/lib/kubelet/pods/ce3360e9-fc71-49d5-b91e-832f129b0913/volumes"
Jul 29 11:43:27.722632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.722592    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723062    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="storage-initializer"
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723082    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="storage-initializer"
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723100    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kube-rbac-proxy"
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723111    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kube-rbac-proxy"
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723166    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container"
Jul 29 11:43:27.723240 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723176    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container"
Jul 29 11:43:27.723575 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723262    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kserve-container"
Jul 29 11:43:27.723575 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.723273    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3360e9-fc71-49d5-b91e-832f129b0913" containerName="kube-rbac-proxy"
Jul 29 11:43:27.726718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.726694    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.731819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.731797    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:43:27.731959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.731799    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:43:27.731959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.731856    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:43:27.731959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.731919    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:43:27.731959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.731891    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:43:27.734412 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.734390    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:43:27.734542 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.734438    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 29 11:43:27.736539 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.736519    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:43:27.815605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.815576    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.815605 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.815609    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.815809 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.815730    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.815809 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.815788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxl9\" (UniqueName: \"kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.815897 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.815883    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.916994 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.916935    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917230 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917004    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxl9\" (UniqueName: \"kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917230 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917059    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917377 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:43:27.917272    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 29 11:43:27.917377 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917377 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:43:27.917358    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls podName:1ff7fe15-a12d-4826-a7b0-5e948d4787b0 nodeName:}" failed. No retries permitted until 2026-07-29 11:43:28.4173346 +0000 UTC m=+4460.319562989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 29 11:43:27.917541 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917378    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917541 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.917914 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.918003 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.917958    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:27.927370 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:27.927344    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxl9\" (UniqueName: \"kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:28.422262 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:28.422226    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:28.422441 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:43:28.422375    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 29 11:43:28.422441 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:43:28.422433    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls podName:1ff7fe15-a12d-4826-a7b0-5e948d4787b0 nodeName:}" failed. No retries permitted until 2026-07-29 11:43:29.422419505 +0000 UTC m=+4461.324647879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 29 11:43:29.430718 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:29.430675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:29.433243 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:29.433218    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-hh94n\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:29.537777 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:29.537740    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:29.671489 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:29.671463    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:43:29.673661 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:43:29.673636    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff7fe15_a12d_4826_a7b0_5e948d4787b0.slice/crio-b1cd9b87930be381f187edcff447728fd0e3a7460db0bd4b1c6b9c7f396feda5 WatchSource:0}: Error finding container b1cd9b87930be381f187edcff447728fd0e3a7460db0bd4b1c6b9c7f396feda5: Status 404 returned error can't find the container with id b1cd9b87930be381f187edcff447728fd0e3a7460db0bd4b1c6b9c7f396feda5
Jul 29 11:43:30.576161 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:30.576112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerStarted","Data":"d50f973eca66db8e78c3c7e8d6fd88a2a4783bfb80a2d5e1fcbf8c6c7daf1fc0"}
Jul 29 11:43:30.576541 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:30.576186    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerStarted","Data":"b1cd9b87930be381f187edcff447728fd0e3a7460db0bd4b1c6b9c7f396feda5"}
Jul 29 11:43:34.594203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:34.594165    2551 generic.go:356] "Generic (PLEG): container finished" podID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerID="d50f973eca66db8e78c3c7e8d6fd88a2a4783bfb80a2d5e1fcbf8c6c7daf1fc0" exitCode=0
Jul 29 11:43:34.594742 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:34.594213    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerDied","Data":"d50f973eca66db8e78c3c7e8d6fd88a2a4783bfb80a2d5e1fcbf8c6c7daf1fc0"}
Jul 29 11:43:34.595525 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:34.595509    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:43:38.617632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.617596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerStarted","Data":"6f1e6e9a4a8cc3fee2ed9f04938c78c4350e6793579d0a61fc4c28f9257da496"}
Jul 29 11:43:38.618028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.617658    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerStarted","Data":"aaab9b0cd3060461eee410effccdfdb36439b22e31f21735905e1be84085ebe9"}
Jul 29 11:43:38.618028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.617927    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:38.618028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.617981    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:38.619438 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.619413    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 29 11:43:38.645298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:38.645249    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podStartSLOduration=3.606752141 podStartE2EDuration="11.645233813s" totalImagesPullingTime="3.576167986s" totalInitContainerRuntime="4.462313686s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:43:38.641085023 +0000 UTC m=+4470.543313420" watchObservedRunningTime="2026-07-29 11:43:38.645233813 +0000 UTC m=+4470.547462279"
Jul 29 11:43:39.621827 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:39.621790    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 29 11:43:44.626532 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:44.626500    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:43:44.627024 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:44.626997    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 29 11:43:54.627395 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:43:54.627365    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:44:07.537315 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:07.537239    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:44:07.537671 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:07.537485    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container" containerID="cri-o://aaab9b0cd3060461eee410effccdfdb36439b22e31f21735905e1be84085ebe9" gracePeriod=30
Jul 29 11:44:07.537671 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:07.537554    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" containerID="cri-o://6f1e6e9a4a8cc3fee2ed9f04938c78c4350e6793579d0a61fc4c28f9257da496" gracePeriod=30
Jul 29 11:44:07.741377 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:07.741351    2551 generic.go:356] "Generic (PLEG): container finished" podID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerID="6f1e6e9a4a8cc3fee2ed9f04938c78c4350e6793579d0a61fc4c28f9257da496" exitCode=2
Jul 29 11:44:07.741479 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:07.741416    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerDied","Data":"6f1e6e9a4a8cc3fee2ed9f04938c78c4350e6793579d0a61fc4c28f9257da496"}
Jul 29 11:44:08.958334 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:08.958305    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:44:08.969139 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:08.969114    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:44:09.622786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:09.622748    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:14.622976 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:14.622939    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:19.622349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:19.622300    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:19.622737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:19.622459    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:44:24.622551 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:24.622507    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:29.622350 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:29.622298    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:34.622176 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:34.622108    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 29 11:44:37.866588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:37.866507    2551 generic.go:356] "Generic (PLEG): container finished" podID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerID="aaab9b0cd3060461eee410effccdfdb36439b22e31f21735905e1be84085ebe9" exitCode=137
Jul 29 11:44:37.866588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:37.866578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerDied","Data":"aaab9b0cd3060461eee410effccdfdb36439b22e31f21735905e1be84085ebe9"}
Jul 29 11:44:38.172106 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.172083    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:44:38.275103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275082    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert\") pod \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") "
Jul 29 11:44:38.275262 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275110    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") pod \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") "
Jul 29 11:44:38.275262 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275149    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config\") pod \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") "
Jul 29 11:44:38.275262 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275183    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location\") pod \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") "
Jul 29 11:44:38.275262 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275238    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbxl9\" (UniqueName: \"kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9\") pod \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\" (UID: \"1ff7fe15-a12d-4826-a7b0-5e948d4787b0\") "
Jul 29 11:44:38.275586 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ff7fe15-a12d-4826-a7b0-5e948d4787b0" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:44:38.275668 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.275568    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ff7fe15-a12d-4826-a7b0-5e948d4787b0" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:44:38.277187 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.277165    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ff7fe15-a12d-4826-a7b0-5e948d4787b0" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:44:38.277369 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.277347    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9" (OuterVolumeSpecName: "kube-api-access-fbxl9") pod "1ff7fe15-a12d-4826-a7b0-5e948d4787b0" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0"). InnerVolumeSpecName "kube-api-access-fbxl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:44:38.291489 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.291465    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ff7fe15-a12d-4826-a7b0-5e948d4787b0" (UID: "1ff7fe15-a12d-4826-a7b0-5e948d4787b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:44:38.376757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.376728    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:44:38.376757 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.376754    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxl9\" (UniqueName: \"kubernetes.io/projected/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-api-access-fbxl9\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:44:38.376894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.376770    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:44:38.376894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.376785    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:44:38.376894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.376799    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ff7fe15-a12d-4826-a7b0-5e948d4787b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:44:38.873208 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.873175    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n" event={"ID":"1ff7fe15-a12d-4826-a7b0-5e948d4787b0","Type":"ContainerDied","Data":"b1cd9b87930be381f187edcff447728fd0e3a7460db0bd4b1c6b9c7f396feda5"}
Jul 29 11:44:38.873554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.873233    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"
Jul 29 11:44:38.873554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.873241    2551 scope.go:117] "RemoveContainer" containerID="6f1e6e9a4a8cc3fee2ed9f04938c78c4350e6793579d0a61fc4c28f9257da496"
Jul 29 11:44:38.882968 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.882950    2551 scope.go:117] "RemoveContainer" containerID="aaab9b0cd3060461eee410effccdfdb36439b22e31f21735905e1be84085ebe9"
Jul 29 11:44:38.890527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.890507    2551 scope.go:117] "RemoveContainer" containerID="d50f973eca66db8e78c3c7e8d6fd88a2a4783bfb80a2d5e1fcbf8c6c7daf1fc0"
Jul 29 11:44:38.891960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.891941    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:44:38.895856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:38.895829    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-hh94n"]
Jul 29 11:44:40.727918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:44:40.727881    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" path="/var/lib/kubelet/pods/1ff7fe15-a12d-4826-a7b0-5e948d4787b0/volumes"
Jul 29 11:46:29.336333 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336304    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336611    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336622    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336637    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="storage-initializer"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336642    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="storage-initializer"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336650    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336655    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336717    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kserve-container"
Jul 29 11:46:29.336729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.336724    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff7fe15-a12d-4826-a7b0-5e948d4787b0" containerName="kube-rbac-proxy"
Jul 29 11:46:29.339708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.339692    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.342359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342332    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 29 11:46:29.342513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342424    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:46:29.342513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342430    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:46:29.342513 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342480    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:46:29.342691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342532    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 29 11:46:29.342691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342545    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:46:29.342855 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.342840    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:46:29.349565 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.349542    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:46:29.422574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.422548    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.422735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.422611    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.422735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.422655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.422735 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.422680    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjh9\" (UniqueName: \"kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.422884 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.422762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.523707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.523628    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.523707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.523673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.523707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.523694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjh9\" (UniqueName: \"kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.523978 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.523803    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.523978 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.523869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.524270 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.524253    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.524570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.524548    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.524632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.524549    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.526150 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.526109    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.531290 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.531267    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjh9\" (UniqueName: \"kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9\") pod \"isvc-xgboost-predictor-7566d65bf7-k8ksp\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.650804 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.650775    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:29.777296 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:29.777272    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:46:29.779737 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:46:29.779712    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a400260_3459_4cf0_8315_a32821624014.slice/crio-ffbe7fbe27d6ae3e197b4dc59a6fdfa4fdc445654c15a500cb8f0dbb961eaf4c WatchSource:0}: Error finding container ffbe7fbe27d6ae3e197b4dc59a6fdfa4fdc445654c15a500cb8f0dbb961eaf4c: Status 404 returned error can't find the container with id ffbe7fbe27d6ae3e197b4dc59a6fdfa4fdc445654c15a500cb8f0dbb961eaf4c
Jul 29 11:46:30.358144 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:30.358085    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerStarted","Data":"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"}
Jul 29 11:46:30.358630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:30.358170    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerStarted","Data":"ffbe7fbe27d6ae3e197b4dc59a6fdfa4fdc445654c15a500cb8f0dbb961eaf4c"}
Jul 29 11:46:34.377118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:34.377085    2551 generic.go:356] "Generic (PLEG): container finished" podID="5a400260-3459-4cf0-8315-a32821624014" containerID="da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e" exitCode=0
Jul 29 11:46:34.377567 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:34.377159    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerDied","Data":"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"}
Jul 29 11:46:53.472095 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.472062    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerStarted","Data":"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"}
Jul 29 11:46:53.472633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.472172    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerStarted","Data":"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"}
Jul 29 11:46:53.472633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.472438    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:53.472633 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.472601    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:46:53.473856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.473830    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:46:53.493052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:53.493012    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podStartSLOduration=2.351894621 podStartE2EDuration="24.492999368s" totalImagesPullingTime="18.394145121s" totalInitContainerRuntime="3.746959626s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:46:53.491516657 +0000 UTC m=+4665.393745053" watchObservedRunningTime="2026-07-29 11:46:53.492999368 +0000 UTC m=+4665.395227765"
Jul 29 11:46:54.477201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:54.477158    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:46:55.481667 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:46:55.481620    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:00.486709 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:00.486633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:47:00.488214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:00.487289    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:10.487300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:10.487251    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:20.487285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:20.487242    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:22.602404 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:22.602369    2551 scope.go:117] "RemoveContainer" containerID="88c06e5440d8f4d6ef84494e305ee3e9e04bf83e31c6df2e7a9411488dbdf0ca"
Jul 29 11:47:22.610078 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:22.610054    2551 scope.go:117] "RemoveContainer" containerID="b38d7a22451a4806d0032d10b57a567cebd52dc7b5f3130e586bfd804c804a25"
Jul 29 11:47:22.617411 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:22.617394    2551 scope.go:117] "RemoveContainer" containerID="95fdfe594f6762b68fc0bb14b7031ba9d5d79d4cfbf84a23e5d29fc26056571e"
Jul 29 11:47:30.487763 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:30.487724    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:40.487814 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:40.487762    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:47:50.488152 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:50.488096    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:47:59.443613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:59.443575    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:47:59.444033 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:59.443848    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" containerID="cri-o://b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4" gracePeriod=30
Jul 29 11:47:59.444033 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:59.443937    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kube-rbac-proxy" containerID="cri-o://0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38" gracePeriod=30
Jul 29 11:47:59.755624 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:59.755596    2551 generic.go:356] "Generic (PLEG): container finished" podID="5a400260-3459-4cf0-8315-a32821624014" containerID="0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38" exitCode=2
Jul 29 11:47:59.755759 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:47:59.755629    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerDied","Data":"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"}
Jul 29 11:48:00.482321 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:00.482273    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.69:8643/healthz\": dial tcp 10.133.0.69:8643: connect: connection refused"
Jul 29 11:48:00.488194 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:00.488161    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 29 11:48:02.387056 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.387031    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:48:02.517871 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.517846    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location\") pod \"5a400260-3459-4cf0-8315-a32821624014\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") "
Jul 29 11:48:02.518023 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.517890    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert\") pod \"5a400260-3459-4cf0-8315-a32821624014\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") "
Jul 29 11:48:02.518023 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.517931    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls\") pod \"5a400260-3459-4cf0-8315-a32821624014\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") "
Jul 29 11:48:02.518023 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.517962    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtjh9\" (UniqueName: \"kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9\") pod \"5a400260-3459-4cf0-8315-a32821624014\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") "
Jul 29 11:48:02.518023 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.517998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config\") pod \"5a400260-3459-4cf0-8315-a32821624014\" (UID: \"5a400260-3459-4cf0-8315-a32821624014\") "
Jul 29 11:48:02.518275 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.518223    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a400260-3459-4cf0-8315-a32821624014" (UID: "5a400260-3459-4cf0-8315-a32821624014"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:48:02.518356 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.518327    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a400260-3459-4cf0-8315-a32821624014" (UID: "5a400260-3459-4cf0-8315-a32821624014"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:48:02.518485 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.518455    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a400260-3459-4cf0-8315-a32821624014" (UID: "5a400260-3459-4cf0-8315-a32821624014"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:48:02.520045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.520023    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a400260-3459-4cf0-8315-a32821624014" (UID: "5a400260-3459-4cf0-8315-a32821624014"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:48:02.520155 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.520045    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9" (OuterVolumeSpecName: "kube-api-access-mtjh9") pod "5a400260-3459-4cf0-8315-a32821624014" (UID: "5a400260-3459-4cf0-8315-a32821624014"). InnerVolumeSpecName "kube-api-access-mtjh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:48:02.618827 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.618795    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a400260-3459-4cf0-8315-a32821624014-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:48:02.618827 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.618824    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjh9\" (UniqueName: \"kubernetes.io/projected/5a400260-3459-4cf0-8315-a32821624014-kube-api-access-mtjh9\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:48:02.618827 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.618834    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:48:02.618998 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.618844    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a400260-3459-4cf0-8315-a32821624014-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:48:02.618998 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.618853    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a400260-3459-4cf0-8315-a32821624014-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:48:02.769616 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.769552    2551 generic.go:356] "Generic (PLEG): container finished" podID="5a400260-3459-4cf0-8315-a32821624014" containerID="b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4" exitCode=0
Jul 29 11:48:02.769616 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.769609    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"
Jul 29 11:48:02.769774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.769635    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerDied","Data":"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"}
Jul 29 11:48:02.769774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.769703    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp" event={"ID":"5a400260-3459-4cf0-8315-a32821624014","Type":"ContainerDied","Data":"ffbe7fbe27d6ae3e197b4dc59a6fdfa4fdc445654c15a500cb8f0dbb961eaf4c"}
Jul 29 11:48:02.769774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.769743    2551 scope.go:117] "RemoveContainer" containerID="0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"
Jul 29 11:48:02.778324 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.778306    2551 scope.go:117] "RemoveContainer" containerID="b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"
Jul 29 11:48:02.785843 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.785822    2551 scope.go:117] "RemoveContainer" containerID="da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"
Jul 29 11:48:02.786785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.786754    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:48:02.790715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.790696    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-k8ksp"]
Jul 29 11:48:02.793510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.793494    2551 scope.go:117] "RemoveContainer" containerID="0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"
Jul 29 11:48:02.793748 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:48:02.793731    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38\": container with ID starting with 0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38 not found: ID does not exist" containerID="0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"
Jul 29 11:48:02.793824 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.793757    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38"} err="failed to get container status \"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38\": rpc error: code = NotFound desc = could not find container \"0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38\": container with ID starting with 0ca27e47e48b41f605db65aa663c6b925ae17114d972d9d688dc5a12e7290b38 not found: ID does not exist"
Jul 29 11:48:02.793824 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.793779    2551 scope.go:117] "RemoveContainer" containerID="b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"
Jul 29 11:48:02.794040 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:48:02.794020    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4\": container with ID starting with b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4 not found: ID does not exist" containerID="b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"
Jul 29 11:48:02.794086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.794047    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4"} err="failed to get container status \"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4\": rpc error: code = NotFound desc = could not find container \"b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4\": container with ID starting with b142bb642176878fa43cc715b5a8f6c9cfda87e36a55cd59c88ebdba44ebb3f4 not found: ID does not exist"
Jul 29 11:48:02.794086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.794070    2551 scope.go:117] "RemoveContainer" containerID="da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"
Jul 29 11:48:02.794324 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:48:02.794307    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e\": container with ID starting with da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e not found: ID does not exist" containerID="da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"
Jul 29 11:48:02.794369 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:02.794329    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e"} err="failed to get container status \"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e\": rpc error: code = NotFound desc = could not find container \"da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e\": container with ID starting with da94547ab1154b07ee18ff1292c59c5783295008129c3f10cc96a6a59850103e not found: ID does not exist"
Jul 29 11:48:04.605806 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.605771    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606097    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kube-rbac-proxy"
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606109    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kube-rbac-proxy"
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606145    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="storage-initializer"
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606154    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="storage-initializer"
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606169    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container"
Jul 29 11:48:04.606224 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606178    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container"
Jul 29 11:48:04.606428 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606236    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kserve-container"
Jul 29 11:48:04.606428 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.606245    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a400260-3459-4cf0-8315-a32821624014" containerName="kube-rbac-proxy"
Jul 29 11:48:04.609440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.609423    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612779    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612790    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612797    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612803    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612828    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612828    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 29 11:48:04.612924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.612897    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:48:04.619554 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.619532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:48:04.729552 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.729516    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a400260-3459-4cf0-8315-a32821624014" path="/var/lib/kubelet/pods/5a400260-3459-4cf0-8315-a32821624014/volumes"
Jul 29 11:48:04.735900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.735869    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.736034 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.735924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.736100 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.736032    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.736100 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.736065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.736227 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.736158    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.836597 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.836571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.836597 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.836597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.836737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.836624    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.836737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.836642    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.836737 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.836678    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.837060 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.837037    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.837390 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.837374    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.837440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.837386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.839019 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.838998    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.844460 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.844440    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:04.921115 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:04.921059    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:05.045440 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:05.045410    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:48:05.048564 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:48:05.048537    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ad58d_e565_4487_b7e8_6db19dafa890.slice/crio-793483d52fb4e0707d5f49610b290d94f1184fd73dfe1c1b65950cfb63d7c9a1 WatchSource:0}: Error finding container 793483d52fb4e0707d5f49610b290d94f1184fd73dfe1c1b65950cfb63d7c9a1: Status 404 returned error can't find the container with id 793483d52fb4e0707d5f49610b290d94f1184fd73dfe1c1b65950cfb63d7c9a1
Jul 29 11:48:05.783833 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:05.783792    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerStarted","Data":"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"}
Jul 29 11:48:05.784328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:05.783867    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerStarted","Data":"793483d52fb4e0707d5f49610b290d94f1184fd73dfe1c1b65950cfb63d7c9a1"}
Jul 29 11:48:08.797474 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:08.797447    2551 generic.go:356] "Generic (PLEG): container finished" podID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerID="a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c" exitCode=0
Jul 29 11:48:08.797779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:08.797492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerDied","Data":"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"}
Jul 29 11:48:09.803276 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:09.803241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerStarted","Data":"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"}
Jul 29 11:48:09.803647 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:09.803305    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerStarted","Data":"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"}
Jul 29 11:48:09.803647 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:09.803495    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:09.823183 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:09.823120    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" podStartSLOduration=2.178604947 podStartE2EDuration="5.823105546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644500599s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:48:09.821843061 +0000 UTC m=+4741.724071461" watchObservedRunningTime="2026-07-29 11:48:09.823105546 +0000 UTC m=+4741.725333943"
Jul 29 11:48:10.807848 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:10.807808    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:16.823606 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:16.823579    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:46.828399 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:46.828367    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:48:54.663076 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:54.663039    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:48:54.663540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:54.663304    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kserve-container" containerID="cri-o://97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75" gracePeriod=30
Jul 29 11:48:54.663540 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:54.663371    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kube-rbac-proxy" containerID="cri-o://c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55" gracePeriod=30
Jul 29 11:48:54.988682 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:54.988646    2551 generic.go:356] "Generic (PLEG): container finished" podID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerID="c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55" exitCode=2
Jul 29 11:48:54.988859 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:54.988721    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerDied","Data":"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"}
Jul 29 11:48:56.812360 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:56.812317    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Jul 29 11:48:59.839527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.839492    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:48:59.843151 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.843121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:48:59.845418 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.845395    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 29 11:48:59.845589 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.845569    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 29 11:48:59.856165 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.856142    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:48:59.941109 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.941078    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:48:59.941327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.941191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259l8\" (UniqueName: \"kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:48:59.941327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.941227    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:48:59.941327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.941257    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:48:59.941327 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:48:59.941283    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042409    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259l8\" (UniqueName: \"kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042447    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042468    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042483    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042518    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.042989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.042963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.043092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.043073    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.043238 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.043220    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.045021 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.045001    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.050306 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.050285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-259l8\" (UniqueName: \"kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.153600 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.153508    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:00.288116 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:00.288067    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:49:00.290974 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:49:00.290942    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10a67f5_0927_479f_8e1b_b995c0c37107.slice/crio-4598d42500d30886a24b1029d7ffaf5c548eb05ff1979ce396e7b3bce4ea6cf1 WatchSource:0}: Error finding container 4598d42500d30886a24b1029d7ffaf5c548eb05ff1979ce396e7b3bce4ea6cf1: Status 404 returned error can't find the container with id 4598d42500d30886a24b1029d7ffaf5c548eb05ff1979ce396e7b3bce4ea6cf1
Jul 29 11:49:01.005308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.005282    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:49:01.015276 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.015243    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerStarted","Data":"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"}
Jul 29 11:49:01.015416 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.015306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerStarted","Data":"4598d42500d30886a24b1029d7ffaf5c548eb05ff1979ce396e7b3bce4ea6cf1"}
Jul 29 11:49:01.017018 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.016993    2551 generic.go:356] "Generic (PLEG): container finished" podID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerID="97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75" exitCode=0
Jul 29 11:49:01.017094 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.017032    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"
Jul 29 11:49:01.017094 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.017059    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerDied","Data":"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"}
Jul 29 11:49:01.017196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.017109    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr" event={"ID":"f67ad58d-e565-4487-b7e8-6db19dafa890","Type":"ContainerDied","Data":"793483d52fb4e0707d5f49610b290d94f1184fd73dfe1c1b65950cfb63d7c9a1"}
Jul 29 11:49:01.017196 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.017163    2551 scope.go:117] "RemoveContainer" containerID="c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"
Jul 29 11:49:01.025054 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.025036    2551 scope.go:117] "RemoveContainer" containerID="97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"
Jul 29 11:49:01.033931 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.033909    2551 scope.go:117] "RemoveContainer" containerID="a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"
Jul 29 11:49:01.042941 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.042924    2551 scope.go:117] "RemoveContainer" containerID="c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"
Jul 29 11:49:01.043226 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:01.043207    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55\": container with ID starting with c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55 not found: ID does not exist" containerID="c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"
Jul 29 11:49:01.043297 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.043235    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55"} err="failed to get container status \"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55\": rpc error: code = NotFound desc = could not find container \"c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55\": container with ID starting with c384cb4c918bc0039662ac7900ac7f00f56860a96b97b5f9ccb856aed5986d55 not found: ID does not exist"
Jul 29 11:49:01.043297 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.043252    2551 scope.go:117] "RemoveContainer" containerID="97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"
Jul 29 11:49:01.043528 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:01.043500    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75\": container with ID starting with 97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75 not found: ID does not exist" containerID="97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"
Jul 29 11:49:01.043574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.043538    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75"} err="failed to get container status \"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75\": rpc error: code = NotFound desc = could not find container \"97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75\": container with ID starting with 97dee088e4242a79dd25561e1808757a752dbb7c96433b27cb4d9f3ec056ba75 not found: ID does not exist"
Jul 29 11:49:01.043574 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.043561    2551 scope.go:117] "RemoveContainer" containerID="a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"
Jul 29 11:49:01.043847 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:01.043829    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c\": container with ID starting with a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c not found: ID does not exist" containerID="a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"
Jul 29 11:49:01.043912 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.043854    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c"} err="failed to get container status \"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c\": rpc error: code = NotFound desc = could not find container \"a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c\": container with ID starting with a437a8ad2cd928f30f4c18b141be92d9fcdaf252e5809c962cb3693e1388879c not found: ID does not exist"
Jul 29 11:49:01.051632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051555    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls\") pod \"f67ad58d-e565-4487-b7e8-6db19dafa890\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") "
Jul 29 11:49:01.051632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051578    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert\") pod \"f67ad58d-e565-4487-b7e8-6db19dafa890\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") "
Jul 29 11:49:01.051632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051606    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk\") pod \"f67ad58d-e565-4487-b7e8-6db19dafa890\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") "
Jul 29 11:49:01.051632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051630    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location\") pod \"f67ad58d-e565-4487-b7e8-6db19dafa890\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") "
Jul 29 11:49:01.051953 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051686    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config\") pod \"f67ad58d-e565-4487-b7e8-6db19dafa890\" (UID: \"f67ad58d-e565-4487-b7e8-6db19dafa890\") "
Jul 29 11:49:01.052295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.051999    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f67ad58d-e565-4487-b7e8-6db19dafa890" (UID: "f67ad58d-e565-4487-b7e8-6db19dafa890"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:01.052613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.052576    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f67ad58d-e565-4487-b7e8-6db19dafa890" (UID: "f67ad58d-e565-4487-b7e8-6db19dafa890"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:49:01.052703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.052619    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f67ad58d-e565-4487-b7e8-6db19dafa890" (UID: "f67ad58d-e565-4487-b7e8-6db19dafa890"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:49:01.053881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.053856    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f67ad58d-e565-4487-b7e8-6db19dafa890" (UID: "f67ad58d-e565-4487-b7e8-6db19dafa890"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:49:01.053984 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.053966    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk" (OuterVolumeSpecName: "kube-api-access-v96zk") pod "f67ad58d-e565-4487-b7e8-6db19dafa890" (UID: "f67ad58d-e565-4487-b7e8-6db19dafa890"). InnerVolumeSpecName "kube-api-access-v96zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:49:01.153162 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.153121    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ad58d-e565-4487-b7e8-6db19dafa890-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:01.153267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.153164    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:01.153267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.153177    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96zk\" (UniqueName: \"kubernetes.io/projected/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-api-access-v96zk\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:01.153267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.153189    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ad58d-e565-4487-b7e8-6db19dafa890-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:01.153267 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.153202    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ad58d-e565-4487-b7e8-6db19dafa890-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:01.338415 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.338380    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:49:01.341792 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:01.341770    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-k9xsr"]
Jul 29 11:49:02.728959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:02.728922    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" path="/var/lib/kubelet/pods/f67ad58d-e565-4487-b7e8-6db19dafa890/volumes"
Jul 29 11:49:04.031754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:04.031722    2551 generic.go:356] "Generic (PLEG): container finished" podID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerID="3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad" exitCode=0
Jul 29 11:49:04.032118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:04.031785    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerDied","Data":"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"}
Jul 29 11:49:05.037926 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:05.037892    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerStarted","Data":"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"}
Jul 29 11:49:05.038366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:05.037955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerStarted","Data":"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"}
Jul 29 11:49:05.038366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:05.038257    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:05.058596 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:05.058550    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" podStartSLOduration=2.471852409 podStartE2EDuration="6.058535671s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586683262s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:49:05.05671752 +0000 UTC m=+4796.958945917" watchObservedRunningTime="2026-07-29 11:49:05.058535671 +0000 UTC m=+4796.960764068"
Jul 29 11:49:06.043065 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:06.043034    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:08.987632 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:08.987604    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:49:08.998380 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:08.998360    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:49:12.052237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:12.052206    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:42.056808 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:42.056770    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:49.914981 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:49.914950    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:49:49.915388 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:49.915172    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kserve-container" containerID="cri-o://bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6" gracePeriod=30
Jul 29 11:49:49.915388 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:49.915251    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kube-rbac-proxy" containerID="cri-o://bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36" gracePeriod=30
Jul 29 11:49:50.239252 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:50.239222    2551 generic.go:356] "Generic (PLEG): container finished" podID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerID="bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36" exitCode=2
Jul 29 11:49:50.239389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:50.239293    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerDied","Data":"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"}
Jul 29 11:49:52.047743 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:52.047704    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 29 11:49:52.053013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:52.052983    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.71:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 29 11:49:54.961563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.961541    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:54.967446 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967427    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls\") pod \"b10a67f5-0927-479f-8e1b-b995c0c37107\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") "
Jul 29 11:49:54.967542 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967480    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location\") pod \"b10a67f5-0927-479f-8e1b-b995c0c37107\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") "
Jul 29 11:49:54.967542 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967537    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-259l8\" (UniqueName: \"kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8\") pod \"b10a67f5-0927-479f-8e1b-b995c0c37107\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") "
Jul 29 11:49:54.967648 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967564    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert\") pod \"b10a67f5-0927-479f-8e1b-b995c0c37107\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") "
Jul 29 11:49:54.967648 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967592    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config\") pod \"b10a67f5-0927-479f-8e1b-b995c0c37107\" (UID: \"b10a67f5-0927-479f-8e1b-b995c0c37107\") "
Jul 29 11:49:54.967894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967833    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b10a67f5-0927-479f-8e1b-b995c0c37107" (UID: "b10a67f5-0927-479f-8e1b-b995c0c37107"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:54.967975 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967903    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b10a67f5-0927-479f-8e1b-b995c0c37107" (UID: "b10a67f5-0927-479f-8e1b-b995c0c37107"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:49:54.968038 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.967976    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b10a67f5-0927-479f-8e1b-b995c0c37107" (UID: "b10a67f5-0927-479f-8e1b-b995c0c37107"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:49:54.969498 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.969480    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b10a67f5-0927-479f-8e1b-b995c0c37107" (UID: "b10a67f5-0927-479f-8e1b-b995c0c37107"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:49:54.969582 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:54.969547    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8" (OuterVolumeSpecName: "kube-api-access-259l8") pod "b10a67f5-0927-479f-8e1b-b995c0c37107" (UID: "b10a67f5-0927-479f-8e1b-b995c0c37107"). InnerVolumeSpecName "kube-api-access-259l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:49:55.068505 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.068436    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-259l8\" (UniqueName: \"kubernetes.io/projected/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-api-access-259l8\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:55.068505 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.068459    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:55.068505 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.068469    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b10a67f5-0927-479f-8e1b-b995c0c37107-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:55.068505 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.068477    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b10a67f5-0927-479f-8e1b-b995c0c37107-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:55.068505 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.068486    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b10a67f5-0927-479f-8e1b-b995c0c37107-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:49:55.087938 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.087911    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:49:55.088259 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088246    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kserve-container"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088263    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kserve-container"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088281    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kserve-container"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088286    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kserve-container"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088295    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088301    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088307 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088309    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="storage-initializer"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088314    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="storage-initializer"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088320    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088325    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088332    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="storage-initializer"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088340    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="storage-initializer"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088394    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088402    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ad58d-e565-4487-b7e8-6db19dafa890" containerName="kserve-container"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088409    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kube-rbac-proxy"
Jul 29 11:49:55.088492 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.088416    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerName="kserve-container"
Jul 29 11:49:55.091629 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.091614    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.093868 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.093848    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:49:55.093963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.093929    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 29 11:49:55.120454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.120433    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:49:55.169303 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.169279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcl9z\" (UniqueName: \"kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.169400 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.169323    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.169400 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.169343    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.169474 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.169427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.169514 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.169499    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.263518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.263494    2551 generic.go:356] "Generic (PLEG): container finished" podID="b10a67f5-0927-479f-8e1b-b995c0c37107" containerID="bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6" exitCode=0
Jul 29 11:49:55.263613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.263544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerDied","Data":"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"}
Jul 29 11:49:55.263613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.263550    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"
Jul 29 11:49:55.263613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.263595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb" event={"ID":"b10a67f5-0927-479f-8e1b-b995c0c37107","Type":"ContainerDied","Data":"4598d42500d30886a24b1029d7ffaf5c548eb05ff1979ce396e7b3bce4ea6cf1"}
Jul 29 11:49:55.263719 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.263624    2551 scope.go:117] "RemoveContainer" containerID="bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"
Jul 29 11:49:55.270046 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270026    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270163 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270054    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcl9z\" (UniqueName: \"kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270163 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270163 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270116    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270467    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270841 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270822    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.270923 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.270860    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.272813 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.272714    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.272882 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.272817    2551 scope.go:117] "RemoveContainer" containerID="bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"
Jul 29 11:49:55.277010 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.276990    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcl9z\" (UniqueName: \"kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-f8fz4\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.280035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.280017    2551 scope.go:117] "RemoveContainer" containerID="3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"
Jul 29 11:49:55.287491 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.287474    2551 scope.go:117] "RemoveContainer" containerID="bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"
Jul 29 11:49:55.287763 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:55.287745    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36\": container with ID starting with bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36 not found: ID does not exist" containerID="bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"
Jul 29 11:49:55.287844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.287769    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36"} err="failed to get container status \"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36\": rpc error: code = NotFound desc = could not find container \"bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36\": container with ID starting with bf18bfb42a54d20c5e1c63d92b464623f123fbbbba192c40bc9e3edbede1ef36 not found: ID does not exist"
Jul 29 11:49:55.287844 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.287797    2551 scope.go:117] "RemoveContainer" containerID="bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"
Jul 29 11:49:55.288058 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:55.288031    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6\": container with ID starting with bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6 not found: ID does not exist" containerID="bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"
Jul 29 11:49:55.288176 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.288067    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6"} err="failed to get container status \"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6\": rpc error: code = NotFound desc = could not find container \"bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6\": container with ID starting with bc8ee16870f5d1fa975445a7c57810175281991ad05750bdae85f3c8dee98ee6 not found: ID does not exist"
Jul 29 11:49:55.288176 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.288091    2551 scope.go:117] "RemoveContainer" containerID="3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"
Jul 29 11:49:55.288354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.288336    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:49:55.288407 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:49:55.288357    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad\": container with ID starting with 3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad not found: ID does not exist" containerID="3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"
Jul 29 11:49:55.288407 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.288380    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad"} err="failed to get container status \"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad\": rpc error: code = NotFound desc = could not find container \"3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad\": container with ID starting with 3059bf608fe95a52cf1e0f406e924c5e11a8bae7ead43071a00c6ec7ad50d1ad not found: ID does not exist"
Jul 29 11:49:55.293935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.293918    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-mtzpb"]
Jul 29 11:49:55.402053 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.401997    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:49:55.529071 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:55.529046    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:49:55.531280 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:49:55.531253    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73d565a_3c44_49a6_980b_7f7662a52b48.slice/crio-495dd9ed3a2ef089ead4d2ab6f870e3a6e81576ab887f536caac8488c712c49e WatchSource:0}: Error finding container 495dd9ed3a2ef089ead4d2ab6f870e3a6e81576ab887f536caac8488c712c49e: Status 404 returned error can't find the container with id 495dd9ed3a2ef089ead4d2ab6f870e3a6e81576ab887f536caac8488c712c49e
Jul 29 11:49:56.269418 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:56.269384    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerStarted","Data":"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"}
Jul 29 11:49:56.269881 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:56.269439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerStarted","Data":"495dd9ed3a2ef089ead4d2ab6f870e3a6e81576ab887f536caac8488c712c49e"}
Jul 29 11:49:56.733146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:49:56.733097    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b10a67f5-0927-479f-8e1b-b995c0c37107" path="/var/lib/kubelet/pods/b10a67f5-0927-479f-8e1b-b995c0c37107/volumes"
Jul 29 11:50:00.287956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:00.287911    2551 generic.go:356] "Generic (PLEG): container finished" podID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerID="87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d" exitCode=0
Jul 29 11:50:00.288408 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:00.287987    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerDied","Data":"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"}
Jul 29 11:50:01.294487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.294456    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerStarted","Data":"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"}
Jul 29 11:50:01.294940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.294523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerStarted","Data":"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"}
Jul 29 11:50:01.294940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.294819    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:50:01.294940 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.294883    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:50:01.296122 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.296098    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:01.318840 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:01.318799    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podStartSLOduration=2.517878492 podStartE2EDuration="6.318787085s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800908593s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:50:01.317481903 +0000 UTC m=+4853.219710299" watchObservedRunningTime="2026-07-29 11:50:01.318787085 +0000 UTC m=+4853.221015481"
Jul 29 11:50:02.298516 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:02.298481    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:07.303183 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:07.303151    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:50:07.303839 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:07.303807    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:17.304581 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:17.304540    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:27.304456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:27.304417    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:37.304019 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:37.303978    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:47.304025 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:47.303987    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:50:57.304320 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:50:57.304286    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:51:05.172698 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:05.172664    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:51:05.173254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:05.172902    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" containerID="cri-o://42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2" gracePeriod=30
Jul 29 11:51:05.173254 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:05.172947    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kube-rbac-proxy" containerID="cri-o://07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da" gracePeriod=30
Jul 29 11:51:05.570627 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:05.570603    2551 generic.go:356] "Generic (PLEG): container finished" podID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerID="07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da" exitCode=2
Jul 29 11:51:05.570774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:05.570675    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerDied","Data":"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"}
Jul 29 11:51:07.299353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:07.299313    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.72:8643/healthz\": dial tcp 10.133.0.72:8643: connect: connection refused"
Jul 29 11:51:07.303732 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:07.303704    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 29 11:51:08.116330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.116308    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:51:08.212203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212177    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location\") pod \"d73d565a-3c44-49a6-980b-7f7662a52b48\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") "
Jul 29 11:51:08.212328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212217    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls\") pod \"d73d565a-3c44-49a6-980b-7f7662a52b48\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") "
Jul 29 11:51:08.212328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212255    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config\") pod \"d73d565a-3c44-49a6-980b-7f7662a52b48\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") "
Jul 29 11:51:08.212328 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212279    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcl9z\" (UniqueName: \"kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z\") pod \"d73d565a-3c44-49a6-980b-7f7662a52b48\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") "
Jul 29 11:51:08.212510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212357    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert\") pod \"d73d565a-3c44-49a6-980b-7f7662a52b48\" (UID: \"d73d565a-3c44-49a6-980b-7f7662a52b48\") "
Jul 29 11:51:08.212626 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212529    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d73d565a-3c44-49a6-980b-7f7662a52b48" (UID: "d73d565a-3c44-49a6-980b-7f7662a52b48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:51:08.212732 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212605    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d73d565a-3c44-49a6-980b-7f7662a52b48" (UID: "d73d565a-3c44-49a6-980b-7f7662a52b48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:51:08.212732 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.212702    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d73d565a-3c44-49a6-980b-7f7662a52b48" (UID: "d73d565a-3c44-49a6-980b-7f7662a52b48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:51:08.214214 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.214194    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d73d565a-3c44-49a6-980b-7f7662a52b48" (UID: "d73d565a-3c44-49a6-980b-7f7662a52b48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:51:08.214354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.214333    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z" (OuterVolumeSpecName: "kube-api-access-wcl9z") pod "d73d565a-3c44-49a6-980b-7f7662a52b48" (UID: "d73d565a-3c44-49a6-980b-7f7662a52b48"). InnerVolumeSpecName "kube-api-access-wcl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:51:08.313711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.313690    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d73d565a-3c44-49a6-980b-7f7662a52b48-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:51:08.313711 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.313709    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d73d565a-3c44-49a6-980b-7f7662a52b48-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:51:08.314012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.313720    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:51:08.314012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.313729    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcl9z\" (UniqueName: \"kubernetes.io/projected/d73d565a-3c44-49a6-980b-7f7662a52b48-kube-api-access-wcl9z\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:51:08.314012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.313737    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d73d565a-3c44-49a6-980b-7f7662a52b48-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:51:08.584824 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.584769    2551 generic.go:356] "Generic (PLEG): container finished" podID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerID="42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2" exitCode=0
Jul 29 11:51:08.584918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.584823    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"
Jul 29 11:51:08.584918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.584844    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerDied","Data":"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"}
Jul 29 11:51:08.584918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.584897    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4" event={"ID":"d73d565a-3c44-49a6-980b-7f7662a52b48","Type":"ContainerDied","Data":"495dd9ed3a2ef089ead4d2ab6f870e3a6e81576ab887f536caac8488c712c49e"}
Jul 29 11:51:08.585031 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.584925    2551 scope.go:117] "RemoveContainer" containerID="07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"
Jul 29 11:51:08.593508 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.593490    2551 scope.go:117] "RemoveContainer" containerID="42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"
Jul 29 11:51:08.600578 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.600560    2551 scope.go:117] "RemoveContainer" containerID="87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"
Jul 29 11:51:08.605847 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.605820    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:51:08.608386 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.608360    2551 scope.go:117] "RemoveContainer" containerID="07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"
Jul 29 11:51:08.608671 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:51:08.608652    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da\": container with ID starting with 07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da not found: ID does not exist" containerID="07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"
Jul 29 11:51:08.608745 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.608679    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da"} err="failed to get container status \"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da\": rpc error: code = NotFound desc = could not find container \"07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da\": container with ID starting with 07af803cd52f0d682d10ea87320790a59a09bf9ec1c1d52fb8d8130f330d66da not found: ID does not exist"
Jul 29 11:51:08.608745 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.608702    2551 scope.go:117] "RemoveContainer" containerID="42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"
Jul 29 11:51:08.608973 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:51:08.608953    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2\": container with ID starting with 42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2 not found: ID does not exist" containerID="42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"
Jul 29 11:51:08.609051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.608979    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2"} err="failed to get container status \"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2\": rpc error: code = NotFound desc = could not find container \"42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2\": container with ID starting with 42c12477ee1d3cb52cca90a7913e0438b4d8b93d5e308416e0e8d62ef0cb71c2 not found: ID does not exist"
Jul 29 11:51:08.609051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.608996    2551 scope.go:117] "RemoveContainer" containerID="87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"
Jul 29 11:51:08.609280 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:51:08.609264    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d\": container with ID starting with 87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d not found: ID does not exist" containerID="87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"
Jul 29 11:51:08.609331 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.609291    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d"} err="failed to get container status \"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d\": rpc error: code = NotFound desc = could not find container \"87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d\": container with ID starting with 87ad679ab9d58bf89f08ad55534890d5f5349dda8eeefa7c41d0213edb52c82d not found: ID does not exist"
Jul 29 11:51:08.609453 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.609434    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-f8fz4"]
Jul 29 11:51:08.728013 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:08.727991    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" path="/var/lib/kubelet/pods/d73d565a-3c44-49a6-980b-7f7662a52b48/volumes"
Jul 29 11:51:10.336856 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.336822    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337156    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kube-rbac-proxy"
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337167    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kube-rbac-proxy"
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337177    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="storage-initializer"
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337182    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="storage-initializer"
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337192    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container"
Jul 29 11:51:10.337237 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337197    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container"
Jul 29 11:51:10.337425 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337245    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kserve-container"
Jul 29 11:51:10.337425 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.337256    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73d565a-3c44-49a6-980b-7f7662a52b48" containerName="kube-rbac-proxy"
Jul 29 11:51:10.341843 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.341826    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.344456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344431    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344459    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344481    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344490    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344436    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344432    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:51:10.344614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.344540    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:51:10.350880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.350859    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:51:10.429680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.429655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.429793 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.429682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zhj\" (UniqueName: \"kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.429793 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.429710    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.429793 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.429736    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.429929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.429812    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.530389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.530508 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530427    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.530593 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530574    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.530634 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zhj\" (UniqueName: \"kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.530689 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530645    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.531028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.530997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.531181 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.531157    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.531260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.531209    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.533025 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.533000    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.541137 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.541105    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zhj\" (UniqueName: \"kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.653232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.653164    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:10.783583 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:10.783552    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:51:10.786573 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:51:10.786547    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf68c54a_b1a7_413a_bab4_d03febe78aa2.slice/crio-b562bd363c82cafa4288cd8dec0fc0d3c6b5d1893cb68b756af73b9d0c76667d WatchSource:0}: Error finding container b562bd363c82cafa4288cd8dec0fc0d3c6b5d1893cb68b756af73b9d0c76667d: Status 404 returned error can't find the container with id b562bd363c82cafa4288cd8dec0fc0d3c6b5d1893cb68b756af73b9d0c76667d
Jul 29 11:51:11.604631 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:11.604597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerStarted","Data":"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"}
Jul 29 11:51:11.605035 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:11.604678    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerStarted","Data":"b562bd363c82cafa4288cd8dec0fc0d3c6b5d1893cb68b756af73b9d0c76667d"}
Jul 29 11:51:14.618276 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:14.618246    2551 generic.go:356] "Generic (PLEG): container finished" podID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerID="b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956" exitCode=0
Jul 29 11:51:14.618614 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:14.618320    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerDied","Data":"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"}
Jul 29 11:51:15.624059 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:15.624026    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerStarted","Data":"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"}
Jul 29 11:51:15.624488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:15.624089    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerStarted","Data":"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"}
Jul 29 11:51:15.624488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:15.624311    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:15.624488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:15.624352    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:15.644246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:15.644182    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podStartSLOduration=1.942408889 podStartE2EDuration="5.644164587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.701755698s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:51:15.643527898 +0000 UTC m=+4927.545756299" watchObservedRunningTime="2026-07-29 11:51:15.644164587 +0000 UTC m=+4927.546392986"
Jul 29 11:51:21.635580 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:21.635551    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:51:51.682089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:51:51.682038    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 29 11:52:01.638542 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:01.638509    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:52:10.437485 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:10.437441    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:52:10.437972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:10.437716    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container" containerID="cri-o://134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a" gracePeriod=30
Jul 29 11:52:10.437972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:10.437735    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kube-rbac-proxy" containerID="cri-o://7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c" gracePeriod=30
Jul 29 11:52:10.859911 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:10.859881    2551 generic.go:356] "Generic (PLEG): container finished" podID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerID="7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c" exitCode=2
Jul 29 11:52:10.860072 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:10.859954    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerDied","Data":"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"}
Jul 29 11:52:11.629787 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:11.629749    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.73:8643/healthz\": dial tcp 10.133.0.73:8643: connect: connection refused"
Jul 29 11:52:12.677281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:12.677233    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.73:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 11:52:15.590158 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.588265    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:52:15.593874 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.593850    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.596358 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.596329    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 29 11:52:15.596630 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.596609    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 29 11:52:15.598748 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.598727    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:52:15.681563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.681536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.681679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.681580    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.681679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.681613    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.681679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.681643    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.681807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.681726    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gl4\" (UniqueName: \"kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782180 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782196    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782226    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782255    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782483 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782329    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gl4\" (UniqueName: \"kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782776    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.782989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.783038 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.782969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.784755 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.784739    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.789997 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.789963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gl4\" (UniqueName: \"kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4\") pod \"isvc-xgboost-v2-predictor-fcc668bc-zhfl5\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:15.904876 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:15.904813    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:16.029074 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.029050    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:52:16.031424 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:52:16.031383    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9150f122_c076_4159_bdeb_2e4fc8585d06.slice/crio-5a9564015452268cbfa1dec71cb4f19e5db79ab23bb37aece0fa2dc24cdcc0d1 WatchSource:0}: Error finding container 5a9564015452268cbfa1dec71cb4f19e5db79ab23bb37aece0fa2dc24cdcc0d1: Status 404 returned error can't find the container with id 5a9564015452268cbfa1dec71cb4f19e5db79ab23bb37aece0fa2dc24cdcc0d1
Jul 29 11:52:16.369077 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.369056    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:52:16.486797 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.486768    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24zhj\" (UniqueName: \"kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj\") pod \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") "
Jul 29 11:52:16.486959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.486809    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location\") pod \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") "
Jul 29 11:52:16.486959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.486831    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert\") pod \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") "
Jul 29 11:52:16.486959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.486859    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config\") pod \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") "
Jul 29 11:52:16.486959 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.486915    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls\") pod \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\" (UID: \"cf68c54a-b1a7-413a-bab4-d03febe78aa2\") "
Jul 29 11:52:16.487232 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.487205    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf68c54a-b1a7-413a-bab4-d03febe78aa2" (UID: "cf68c54a-b1a7-413a-bab4-d03febe78aa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:52:16.487274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.487246    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf68c54a-b1a7-413a-bab4-d03febe78aa2" (UID: "cf68c54a-b1a7-413a-bab4-d03febe78aa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:52:16.487310 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.487265    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf68c54a-b1a7-413a-bab4-d03febe78aa2" (UID: "cf68c54a-b1a7-413a-bab4-d03febe78aa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:52:16.488806 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.488786    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj" (OuterVolumeSpecName: "kube-api-access-24zhj") pod "cf68c54a-b1a7-413a-bab4-d03febe78aa2" (UID: "cf68c54a-b1a7-413a-bab4-d03febe78aa2"). InnerVolumeSpecName "kube-api-access-24zhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:52:16.488885 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.488854    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf68c54a-b1a7-413a-bab4-d03febe78aa2" (UID: "cf68c54a-b1a7-413a-bab4-d03febe78aa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:52:16.587936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.587910    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:52:16.587936 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.587936    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:52:16.588081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.587951    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:52:16.588081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.587963    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf68c54a-b1a7-413a-bab4-d03febe78aa2-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:52:16.588081 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.587976    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zhj\" (UniqueName: \"kubernetes.io/projected/cf68c54a-b1a7-413a-bab4-d03febe78aa2-kube-api-access-24zhj\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:52:16.887837 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.887808    2551 generic.go:356] "Generic (PLEG): container finished" podID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerID="134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a" exitCode=0
Jul 29 11:52:16.888242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.887863    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"
Jul 29 11:52:16.888242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.887903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerDied","Data":"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"}
Jul 29 11:52:16.888242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.887964    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p" event={"ID":"cf68c54a-b1a7-413a-bab4-d03febe78aa2","Type":"ContainerDied","Data":"b562bd363c82cafa4288cd8dec0fc0d3c6b5d1893cb68b756af73b9d0c76667d"}
Jul 29 11:52:16.888242 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.887994    2551 scope.go:117] "RemoveContainer" containerID="7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"
Jul 29 11:52:16.889960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.889909    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerStarted","Data":"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"}
Jul 29 11:52:16.890099 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.889965    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerStarted","Data":"5a9564015452268cbfa1dec71cb4f19e5db79ab23bb37aece0fa2dc24cdcc0d1"}
Jul 29 11:52:16.897029 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.896999    2551 scope.go:117] "RemoveContainer" containerID="134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"
Jul 29 11:52:16.907025 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.907001    2551 scope.go:117] "RemoveContainer" containerID="b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"
Jul 29 11:52:16.915598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.915582    2551 scope.go:117] "RemoveContainer" containerID="7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"
Jul 29 11:52:16.915879 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:52:16.915854    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c\": container with ID starting with 7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c not found: ID does not exist" containerID="7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"
Jul 29 11:52:16.915935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.915887    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c"} err="failed to get container status \"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c\": rpc error: code = NotFound desc = could not find container \"7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c\": container with ID starting with 7817c88f59d80668abce291d8fda1e05829aad6b62d8bab2bc16fca5fdf4de1c not found: ID does not exist"
Jul 29 11:52:16.915935 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.915903    2551 scope.go:117] "RemoveContainer" containerID="134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"
Jul 29 11:52:16.916169 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:52:16.916150    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a\": container with ID starting with 134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a not found: ID does not exist" containerID="134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"
Jul 29 11:52:16.916245 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.916182    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a"} err="failed to get container status \"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a\": rpc error: code = NotFound desc = could not find container \"134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a\": container with ID starting with 134ef630cc4d8431b59f03a8ea7ba7c298ab9a24b023c7783a780cc324a9e02a not found: ID does not exist"
Jul 29 11:52:16.916245 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.916205    2551 scope.go:117] "RemoveContainer" containerID="b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"
Jul 29 11:52:16.916490 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:52:16.916473    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956\": container with ID starting with b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956 not found: ID does not exist" containerID="b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"
Jul 29 11:52:16.916550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.916497    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956"} err="failed to get container status \"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956\": rpc error: code = NotFound desc = could not find container \"b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956\": container with ID starting with b9f9eba1aff1d72477610c6066bbfb1b94d6961ff440519bbf12d97de416c956 not found: ID does not exist"
Jul 29 11:52:16.923482 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.923462    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:52:16.929473 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:16.929437    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-7z99p"]
Jul 29 11:52:18.728976 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:18.728941    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" path="/var/lib/kubelet/pods/cf68c54a-b1a7-413a-bab4-d03febe78aa2/volumes"
Jul 29 11:52:19.906423 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:19.906394    2551 generic.go:356] "Generic (PLEG): container finished" podID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerID="f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88" exitCode=0
Jul 29 11:52:19.906812 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:19.906430    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerDied","Data":"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"}
Jul 29 11:52:20.912177 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.912124    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerStarted","Data":"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"}
Jul 29 11:52:20.912549 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.912203    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerStarted","Data":"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"}
Jul 29 11:52:20.918256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.913633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:20.918256 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.913754    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:20.918487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.918455    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:20.935679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:20.935631    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podStartSLOduration=2.213611023 podStartE2EDuration="5.9356171s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.722006077s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:52:20.934235107 +0000 UTC m=+4992.836463498" watchObservedRunningTime="2026-07-29 11:52:20.9356171 +0000 UTC m=+4992.837845567"
Jul 29 11:52:21.916588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:21.916543    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:22.921121 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:22.921090    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:27.925680 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:27.925647    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:52:27.926285 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:27.926257    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:37.927043 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:37.926999    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:47.926504 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:47.926466    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:52:57.926518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:52:57.926482    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:53:07.926637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:07.926535    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:53:17.927317 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:17.927275    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:53:25.693779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:25.693741    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:53:25.694392 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:25.694035    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" containerID="cri-o://a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307" gracePeriod=30
Jul 29 11:53:25.694392 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:25.694098    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kube-rbac-proxy" containerID="cri-o://e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5" gracePeriod=30
Jul 29 11:53:26.187308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:26.187275    2551 generic.go:356] "Generic (PLEG): container finished" podID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerID="e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5" exitCode=2
Jul 29 11:53:26.187460 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:26.187321    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerDied","Data":"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"}
Jul 29 11:53:27.922146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:27.922096    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.74:8643/healthz\": dial tcp 10.133.0.74:8643: connect: connection refused"
Jul 29 11:53:27.926375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:27.926338    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 29 11:53:28.541443 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.541421    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:53:28.594771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.594747    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location\") pod \"9150f122-c076-4159-bdeb-2e4fc8585d06\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") "
Jul 29 11:53:28.594771 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.594775    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8gl4\" (UniqueName: \"kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4\") pod \"9150f122-c076-4159-bdeb-2e4fc8585d06\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") "
Jul 29 11:53:28.594963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.594793    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert\") pod \"9150f122-c076-4159-bdeb-2e4fc8585d06\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") "
Jul 29 11:53:28.594963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.594814    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config\") pod \"9150f122-c076-4159-bdeb-2e4fc8585d06\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") "
Jul 29 11:53:28.594963 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.594900    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls\") pod \"9150f122-c076-4159-bdeb-2e4fc8585d06\" (UID: \"9150f122-c076-4159-bdeb-2e4fc8585d06\") "
Jul 29 11:53:28.595159 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.595068    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9150f122-c076-4159-bdeb-2e4fc8585d06" (UID: "9150f122-c076-4159-bdeb-2e4fc8585d06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:28.595225 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.595160    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9150f122-c076-4159-bdeb-2e4fc8585d06" (UID: "9150f122-c076-4159-bdeb-2e4fc8585d06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:53:28.595225 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.595175    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9150f122-c076-4159-bdeb-2e4fc8585d06-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:53:28.595225 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.595190    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9150f122-c076-4159-bdeb-2e4fc8585d06" (UID: "9150f122-c076-4159-bdeb-2e4fc8585d06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:53:28.596721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.596701    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4" (OuterVolumeSpecName: "kube-api-access-p8gl4") pod "9150f122-c076-4159-bdeb-2e4fc8585d06" (UID: "9150f122-c076-4159-bdeb-2e4fc8585d06"). InnerVolumeSpecName "kube-api-access-p8gl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:53:28.596819 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.596735    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9150f122-c076-4159-bdeb-2e4fc8585d06" (UID: "9150f122-c076-4159-bdeb-2e4fc8585d06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:53:28.695658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.695595    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gl4\" (UniqueName: \"kubernetes.io/projected/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-api-access-p8gl4\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:53:28.695658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.695623    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:53:28.695658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.695637    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9150f122-c076-4159-bdeb-2e4fc8585d06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:53:28.695658 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:28.695651    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9150f122-c076-4159-bdeb-2e4fc8585d06-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:53:29.201373 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.201345    2551 generic.go:356] "Generic (PLEG): container finished" podID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerID="a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307" exitCode=0
Jul 29 11:53:29.201774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.201378    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerDied","Data":"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"}
Jul 29 11:53:29.201774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.201407    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"
Jul 29 11:53:29.201774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.201430    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5" event={"ID":"9150f122-c076-4159-bdeb-2e4fc8585d06","Type":"ContainerDied","Data":"5a9564015452268cbfa1dec71cb4f19e5db79ab23bb37aece0fa2dc24cdcc0d1"}
Jul 29 11:53:29.201774 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.201467    2551 scope.go:117] "RemoveContainer" containerID="e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"
Jul 29 11:53:29.209518 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.209502    2551 scope.go:117] "RemoveContainer" containerID="a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"
Jul 29 11:53:29.216543 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.216527    2551 scope.go:117] "RemoveContainer" containerID="f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"
Jul 29 11:53:29.221928 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.221906    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:53:29.224203 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.224184    2551 scope.go:117] "RemoveContainer" containerID="e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"
Jul 29 11:53:29.224465 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:53:29.224448    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5\": container with ID starting with e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5 not found: ID does not exist" containerID="e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"
Jul 29 11:53:29.224534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.224472    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5"} err="failed to get container status \"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5\": rpc error: code = NotFound desc = could not find container \"e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5\": container with ID starting with e18fa1b8172c6f8e529c0b75950f42aba63eddb4efdb18cd34108cfe922bcbc5 not found: ID does not exist"
Jul 29 11:53:29.224534 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.224487    2551 scope.go:117] "RemoveContainer" containerID="a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"
Jul 29 11:53:29.224755 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:53:29.224740    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307\": container with ID starting with a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307 not found: ID does not exist" containerID="a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"
Jul 29 11:53:29.224796 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.224760    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307"} err="failed to get container status \"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307\": rpc error: code = NotFound desc = could not find container \"a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307\": container with ID starting with a4b7e63b4a8e47afb082170b9cea03b489343d08f504033d47108ed148c8b307 not found: ID does not exist"
Jul 29 11:53:29.224796 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.224775    2551 scope.go:117] "RemoveContainer" containerID="f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"
Jul 29 11:53:29.225010 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:53:29.224996    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88\": container with ID starting with f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88 not found: ID does not exist" containerID="f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"
Jul 29 11:53:29.225061 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.225013    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88"} err="failed to get container status \"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88\": rpc error: code = NotFound desc = could not find container \"f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88\": container with ID starting with f83f5a07527968ed475315f36b2712a0738a1e6e7f65198e25868164caa83b88 not found: ID does not exist"
Jul 29 11:53:29.225468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:29.225451    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-zhfl5"]
Jul 29 11:53:30.728300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.728264    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" path="/var/lib/kubelet/pods/9150f122-c076-4159-bdeb-2e4fc8585d06/volumes"
Jul 29 11:53:30.854803 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.854767    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:53:30.855107 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855093    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="storage-initializer"
Jul 29 11:53:30.855182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855108    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="storage-initializer"
Jul 29 11:53:30.855182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855138    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855147    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855165    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container"
Jul 29 11:53:30.855182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855174    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855184    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="storage-initializer"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855193    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="storage-initializer"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855200    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855206    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855217    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855222    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855278    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855286    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kube-rbac-proxy"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855293    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf68c54a-b1a7-413a-bab4-d03febe78aa2" containerName="kserve-container"
Jul 29 11:53:30.855340 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.855298    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9150f122-c076-4159-bdeb-2e4fc8585d06" containerName="kserve-container"
Jul 29 11:53:30.859702 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.859683    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:30.862061 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862043    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 29 11:53:30.862191 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862164    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:53:30.862269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862163    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 29 11:53:30.862269 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862208    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:53:30.862525 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862511    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 29 11:53:30.862585 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.862512    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:53:30.870029 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.870006    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:53:30.912625 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.912596    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:30.912721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.912640    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89vs\" (UniqueName: \"kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:30.912721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.912674    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:30.912721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:30.912710    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.013502 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.013446    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.013502 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.013481    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.013637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.013523    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89vs\" (UniqueName: \"kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.013637 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.013571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.014008 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.013987    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.014261 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.014244    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.015877 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.015859    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.020956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.020934    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89vs\" (UniqueName: \"kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs\") pod \"isvc-sklearn-s3-predictor-6988b47874-zvhk6\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.170055 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.170034    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:31.300946 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:31.300923    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:53:31.303008 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:53:31.302979    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329c9f6d_0a91_4bd7_9621_566c1a028fbc.slice/crio-cbb333eeb18d50cbaff00a238050a4fc29ca221adab055dd0313f2c2a569a5f1 WatchSource:0}: Error finding container cbb333eeb18d50cbaff00a238050a4fc29ca221adab055dd0313f2c2a569a5f1: Status 404 returned error can't find the container with id cbb333eeb18d50cbaff00a238050a4fc29ca221adab055dd0313f2c2a569a5f1
Jul 29 11:53:32.216664 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:32.216624    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerStarted","Data":"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"}
Jul 29 11:53:32.217181 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:32.216684    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerStarted","Data":"cbb333eeb18d50cbaff00a238050a4fc29ca221adab055dd0313f2c2a569a5f1"}
Jul 29 11:53:33.225080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:33.225051    2551 generic.go:356] "Generic (PLEG): container finished" podID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerID="99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c" exitCode=0
Jul 29 11:53:33.225570 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:33.225157    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerDied","Data":"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"}
Jul 29 11:53:34.232003 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:34.231971    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerStarted","Data":"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"}
Jul 29 11:53:34.232450 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:34.232024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerStarted","Data":"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"}
Jul 29 11:53:34.232450 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:34.232095    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:34.256616 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:34.253570    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podStartSLOduration=3.192571845 podStartE2EDuration="4.253552566s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.060980721s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:53:34.249696129 +0000 UTC m=+5066.151924528" watchObservedRunningTime="2026-07-29 11:53:34.253552566 +0000 UTC m=+5066.155780968"
Jul 29 11:53:35.236566 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:35.236537    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:35.237919 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:35.237887    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:53:36.240083 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:36.240043    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:53:41.244850 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:41.244817    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:53:41.245444 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:41.245416    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:53:51.246182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:53:51.246122    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:01.245657 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:01.245617    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:09.017566 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:09.017541    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:54:09.027457 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:09.027438    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:54:11.245375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:11.245339    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:21.245469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:21.245432    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:31.246366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:31.246328    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:41.246308 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:41.246271    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:54:50.957074 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:50.957040    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:54:50.957476 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:50.957267    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" containerID="cri-o://5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b" gracePeriod=30
Jul 29 11:54:50.957476 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:50.957305    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kube-rbac-proxy" containerID="cri-o://7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab" gracePeriod=30
Jul 29 11:54:51.093722 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.093691    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:54:51.096956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.096939    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.099295 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.099277    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:54:51.099500 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.099485    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 29 11:54:51.099597 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.099533    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 29 11:54:51.108077 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.108056    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:54:51.169172 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.169151    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6cc7\" (UniqueName: \"kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.169271 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.169188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.169271 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.169253    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.169349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.169302    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.169349 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.169328    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.241298 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.241238    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.75:8643/healthz\": dial tcp 10.133.0.75:8643: connect: connection refused"
Jul 29 11:54:51.245649 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.245621    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.75:8080: connect: connection refused"
Jul 29 11:54:51.270116 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270212 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270212 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270292 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270216    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270292 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6cc7\" (UniqueName: \"kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270382 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:54:51.270342    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 29 11:54:51.270434 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:54:51.270398    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls podName:e888f909-2248-428a-8f77-118a6ed96366 nodeName:}" failed. No retries permitted until 2026-07-29 11:54:51.770377902 +0000 UTC m=+5143.672606289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" (UID: "e888f909-2248-428a-8f77-118a6ed96366") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 29 11:54:51.270524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270501    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270876 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.270924 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.270889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.280622 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.280595    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6cc7\" (UniqueName: \"kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.567011 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.566942    2551 generic.go:356] "Generic (PLEG): container finished" podID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerID="7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab" exitCode=2
Jul 29 11:54:51.567138 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.567014    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerDied","Data":"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"}
Jul 29 11:54:51.774324 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.774294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:51.776537 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:51.776517    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:52.006607 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:52.006573    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:52.134810 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:52.134781    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:54:52.136494 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:54:52.136468    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode888f909_2248_428a_8f77_118a6ed96366.slice/crio-84490d447223543e7b2684b3c42794141a46ea0c14e9ef9ceaf59fa008666f28 WatchSource:0}: Error finding container 84490d447223543e7b2684b3c42794141a46ea0c14e9ef9ceaf59fa008666f28: Status 404 returned error can't find the container with id 84490d447223543e7b2684b3c42794141a46ea0c14e9ef9ceaf59fa008666f28
Jul 29 11:54:52.572670 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:52.572636    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerStarted","Data":"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"}
Jul 29 11:54:52.572837 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:52.572693    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerStarted","Data":"84490d447223543e7b2684b3c42794141a46ea0c14e9ef9ceaf59fa008666f28"}
Jul 29 11:54:53.578271 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:53.578194    2551 generic.go:356] "Generic (PLEG): container finished" podID="e888f909-2248-428a-8f77-118a6ed96366" containerID="97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171" exitCode=0
Jul 29 11:54:53.578271 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:53.578268    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerDied","Data":"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"}
Jul 29 11:54:54.586753 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:54.586710    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerStarted","Data":"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"}
Jul 29 11:54:54.587204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:54.586777    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerStarted","Data":"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"}
Jul 29 11:54:54.587204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:54.586926    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:54.606123 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:54.606076    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podStartSLOduration=2.59598276 podStartE2EDuration="3.606060851s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.010078091s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:54:54.605743465 +0000 UTC m=+5146.507971861" watchObservedRunningTime="2026-07-29 11:54:54.606060851 +0000 UTC m=+5146.508289248"
Jul 29 11:54:55.201847 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.201822    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:54:55.305853 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.305790    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g89vs\" (UniqueName: \"kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs\") pod \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") "
Jul 29 11:54:55.305972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.305858    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config\") pod \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") "
Jul 29 11:54:55.305972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.305908    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location\") pod \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") "
Jul 29 11:54:55.305972 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.305944    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls\") pod \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\" (UID: \"329c9f6d-0a91-4bd7-9621-566c1a028fbc\") "
Jul 29 11:54:55.306236 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.306213    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "329c9f6d-0a91-4bd7-9621-566c1a028fbc" (UID: "329c9f6d-0a91-4bd7-9621-566c1a028fbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:54:55.306301 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.306260    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "329c9f6d-0a91-4bd7-9621-566c1a028fbc" (UID: "329c9f6d-0a91-4bd7-9621-566c1a028fbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:54:55.307813 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.307795    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs" (OuterVolumeSpecName: "kube-api-access-g89vs") pod "329c9f6d-0a91-4bd7-9621-566c1a028fbc" (UID: "329c9f6d-0a91-4bd7-9621-566c1a028fbc"). InnerVolumeSpecName "kube-api-access-g89vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:54:55.307911 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.307890    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "329c9f6d-0a91-4bd7-9621-566c1a028fbc" (UID: "329c9f6d-0a91-4bd7-9621-566c1a028fbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:54:55.407063 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.407040    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329c9f6d-0a91-4bd7-9621-566c1a028fbc-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:54:55.407063 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.407062    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89vs\" (UniqueName: \"kubernetes.io/projected/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-api-access-g89vs\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:54:55.407211 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.407072    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:54:55.407211 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.407081    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/329c9f6d-0a91-4bd7-9621-566c1a028fbc-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:54:55.593628 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.593552    2551 generic.go:356] "Generic (PLEG): container finished" podID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerID="5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b" exitCode=0
Jul 29 11:54:55.593628 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.593602    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"
Jul 29 11:54:55.594017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.593641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerDied","Data":"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"}
Jul 29 11:54:55.594017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.593706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6" event={"ID":"329c9f6d-0a91-4bd7-9621-566c1a028fbc","Type":"ContainerDied","Data":"cbb333eeb18d50cbaff00a238050a4fc29ca221adab055dd0313f2c2a569a5f1"}
Jul 29 11:54:55.594017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.593745    2551 scope.go:117] "RemoveContainer" containerID="7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"
Jul 29 11:54:55.594195 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.594148    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:54:55.595580 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.595548    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:54:55.602687 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.602672    2551 scope.go:117] "RemoveContainer" containerID="5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"
Jul 29 11:54:55.609692 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.609670    2551 scope.go:117] "RemoveContainer" containerID="99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"
Jul 29 11:54:55.615277 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.615254    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:54:55.616943 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.616924    2551 scope.go:117] "RemoveContainer" containerID="7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"
Jul 29 11:54:55.617254 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:54:55.617232    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab\": container with ID starting with 7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab not found: ID does not exist" containerID="7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"
Jul 29 11:54:55.617359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.617264    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab"} err="failed to get container status \"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab\": rpc error: code = NotFound desc = could not find container \"7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab\": container with ID starting with 7e9f826686666a9eb221a58ee2402de156c8ad724c32df776057b56c556fa4ab not found: ID does not exist"
Jul 29 11:54:55.617359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.617287    2551 scope.go:117] "RemoveContainer" containerID="5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"
Jul 29 11:54:55.617682 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:54:55.617657    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b\": container with ID starting with 5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b not found: ID does not exist" containerID="5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"
Jul 29 11:54:55.617754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.617692    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b"} err="failed to get container status \"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b\": rpc error: code = NotFound desc = could not find container \"5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b\": container with ID starting with 5cb98a5f1ec093c939ed0998e58e29944c25c6246ef4fea415108e29510fac6b not found: ID does not exist"
Jul 29 11:54:55.617754 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.617716    2551 scope.go:117] "RemoveContainer" containerID="99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"
Jul 29 11:54:55.617993 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:54:55.617974    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c\": container with ID starting with 99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c not found: ID does not exist" containerID="99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"
Jul 29 11:54:55.618066 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.617999    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c"} err="failed to get container status \"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c\": rpc error: code = NotFound desc = could not find container \"99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c\": container with ID starting with 99d8ed9dfbb435325d210c38211d277be32bd6c04f6bc2a0f4d7d853176a7b0c not found: ID does not exist"
Jul 29 11:54:55.619405 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:55.619387    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6988b47874-zvhk6"]
Jul 29 11:54:56.598913 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:56.598878    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:54:56.728329 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:54:56.728302    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" path="/var/lib/kubelet/pods/329c9f6d-0a91-4bd7-9621-566c1a028fbc/volumes"
Jul 29 11:55:01.604434 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:01.604403    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:55:01.604926 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:01.604887    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:55:11.605698 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:11.605660    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:55:21.605855 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:21.605817    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:55:31.605226 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:31.605187    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:55:41.604990 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:41.604949    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:55:51.605384 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:55:51.605344    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:56:01.605274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:01.605193    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:56:11.605894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:11.605859    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:56:21.133577 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.133536    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:56:21.134150 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.133806    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" containerID="cri-o://6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9" gracePeriod=30
Jul 29 11:56:21.134150 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.133867    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kube-rbac-proxy" containerID="cri-o://b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d" gracePeriod=30
Jul 29 11:56:21.599654 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.599614    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.76:8643/healthz\": dial tcp 10.133.0.76:8643: connect: connection refused"
Jul 29 11:56:21.605822 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.605791    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.76:8080: connect: connection refused"
Jul 29 11:56:21.955900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.955877    2551 generic.go:356] "Generic (PLEG): container finished" podID="e888f909-2248-428a-8f77-118a6ed96366" containerID="b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d" exitCode=2
Jul 29 11:56:21.956051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:21.955907    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerDied","Data":"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"}
Jul 29 11:56:22.263041 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.262962    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:22.263488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263451    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kube-rbac-proxy"
Jul 29 11:56:22.263488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263469    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kube-rbac-proxy"
Jul 29 11:56:22.263488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263484    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container"
Jul 29 11:56:22.263652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263492    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container"
Jul 29 11:56:22.263652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263516    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="storage-initializer"
Jul 29 11:56:22.263652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263527    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="storage-initializer"
Jul 29 11:56:22.263652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263616    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kserve-container"
Jul 29 11:56:22.263652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.263632    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="329c9f6d-0a91-4bd7-9621-566c1a028fbc" containerName="kube-rbac-proxy"
Jul 29 11:56:22.266785 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.266763    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.269080 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.269056    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 29 11:56:22.269201 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.269097    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 29 11:56:22.274589 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.274566    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:22.435721 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.435687    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.435880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.435742    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.435880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.435775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.435880 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.435815    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87wd\" (UniqueName: \"kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.536482 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.536424    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.536482 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.536465    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.536653 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.536487    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.536653 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.536514    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87wd\" (UniqueName: \"kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.536865 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.536846    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.537208 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.537190    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.538804 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.538785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.544967 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.544940    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87wd\" (UniqueName: \"kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.578005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.577979    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:22.702799 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.702740    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:22.705083 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:56:22.705057    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd59370_bab5_4f03_8199_b30ee99671cb.slice/crio-96225962b17d02e9e17124e85048428f10d367e1d80573e7a495b1ed7614c6ca WatchSource:0}: Error finding container 96225962b17d02e9e17124e85048428f10d367e1d80573e7a495b1ed7614c6ca: Status 404 returned error can't find the container with id 96225962b17d02e9e17124e85048428f10d367e1d80573e7a495b1ed7614c6ca
Jul 29 11:56:22.962235 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.962200    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerStarted","Data":"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"}
Jul 29 11:56:22.962379 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:22.962262    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerStarted","Data":"96225962b17d02e9e17124e85048428f10d367e1d80573e7a495b1ed7614c6ca"}
Jul 29 11:56:25.176820 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.176793    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:56:25.259392 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259333    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location\") pod \"e888f909-2248-428a-8f77-118a6ed96366\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") "
Jul 29 11:56:25.259392 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259371    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") pod \"e888f909-2248-428a-8f77-118a6ed96366\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") "
Jul 29 11:56:25.259563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259415    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6cc7\" (UniqueName: \"kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7\") pod \"e888f909-2248-428a-8f77-118a6ed96366\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") "
Jul 29 11:56:25.259563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259455    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config\") pod \"e888f909-2248-428a-8f77-118a6ed96366\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") "
Jul 29 11:56:25.259563 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259481    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert\") pod \"e888f909-2248-428a-8f77-118a6ed96366\" (UID: \"e888f909-2248-428a-8f77-118a6ed96366\") "
Jul 29 11:56:25.259731 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259638    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e888f909-2248-428a-8f77-118a6ed96366" (UID: "e888f909-2248-428a-8f77-118a6ed96366"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:25.259888 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259858    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e888f909-2248-428a-8f77-118a6ed96366" (UID: "e888f909-2248-428a-8f77-118a6ed96366"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:56:25.259956 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.259900    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e888f909-2248-428a-8f77-118a6ed96366" (UID: "e888f909-2248-428a-8f77-118a6ed96366"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:56:25.261454 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.261435    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e888f909-2248-428a-8f77-118a6ed96366" (UID: "e888f909-2248-428a-8f77-118a6ed96366"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:56:25.261524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.261475    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7" (OuterVolumeSpecName: "kube-api-access-f6cc7") pod "e888f909-2248-428a-8f77-118a6ed96366" (UID: "e888f909-2248-428a-8f77-118a6ed96366"). InnerVolumeSpecName "kube-api-access-f6cc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:56:25.360495 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.360472    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:25.360495 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.360492    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e888f909-2248-428a-8f77-118a6ed96366-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:25.360620 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.360501    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e888f909-2248-428a-8f77-118a6ed96366-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:25.360620 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.360510    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e888f909-2248-428a-8f77-118a6ed96366-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:25.360620 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.360520    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6cc7\" (UniqueName: \"kubernetes.io/projected/e888f909-2248-428a-8f77-118a6ed96366-kube-api-access-f6cc7\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:25.979481 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.979445    2551 generic.go:356] "Generic (PLEG): container finished" podID="e888f909-2248-428a-8f77-118a6ed96366" containerID="6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9" exitCode=0
Jul 29 11:56:25.979606 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.979504    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"
Jul 29 11:56:25.979606 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.979538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerDied","Data":"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"}
Jul 29 11:56:25.979691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.979603    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp" event={"ID":"e888f909-2248-428a-8f77-118a6ed96366","Type":"ContainerDied","Data":"84490d447223543e7b2684b3c42794141a46ea0c14e9ef9ceaf59fa008666f28"}
Jul 29 11:56:25.979691 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.979639    2551 scope.go:117] "RemoveContainer" containerID="b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"
Jul 29 11:56:25.989300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:25.989280    2551 scope.go:117] "RemoveContainer" containerID="6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"
Jul 29 11:56:26.000050 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.000029    2551 scope.go:117] "RemoveContainer" containerID="97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"
Jul 29 11:56:26.001330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.001310    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:56:26.004306 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.004284    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-67fbf6b78b-vgctp"]
Jul 29 11:56:26.008424 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.008406    2551 scope.go:117] "RemoveContainer" containerID="b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"
Jul 29 11:56:26.008705 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:26.008691    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d\": container with ID starting with b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d not found: ID does not exist" containerID="b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"
Jul 29 11:56:26.008755 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.008711    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d"} err="failed to get container status \"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d\": rpc error: code = NotFound desc = could not find container \"b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d\": container with ID starting with b2ce6fa0e6b469be94ebd7d2ddc17cd628e4219b5c63fbf6ab31b3e46fd4ce6d not found: ID does not exist"
Jul 29 11:56:26.008755 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.008728    2551 scope.go:117] "RemoveContainer" containerID="6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"
Jul 29 11:56:26.009000 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:26.008978    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9\": container with ID starting with 6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9 not found: ID does not exist" containerID="6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"
Jul 29 11:56:26.009045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.009006    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9"} err="failed to get container status \"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9\": rpc error: code = NotFound desc = could not find container \"6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9\": container with ID starting with 6b94c2d2e2227b1e3226d714ec05382319ec0de4c2aa322626966864000ad2c9 not found: ID does not exist"
Jul 29 11:56:26.009045 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.009031    2551 scope.go:117] "RemoveContainer" containerID="97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"
Jul 29 11:56:26.009277 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:26.009262    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171\": container with ID starting with 97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171 not found: ID does not exist" containerID="97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"
Jul 29 11:56:26.009325 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.009279    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171"} err="failed to get container status \"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171\": rpc error: code = NotFound desc = could not find container \"97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171\": container with ID starting with 97d28a1e64a856742cc70e9d1280c789379bdb15fdee6ccbc22920014bc19171 not found: ID does not exist"
Jul 29 11:56:26.728992 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:26.728956    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e888f909-2248-428a-8f77-118a6ed96366" path="/var/lib/kubelet/pods/e888f909-2248-428a-8f77-118a6ed96366/volumes"
Jul 29 11:56:27.990067 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:27.990044    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/0.log"
Jul 29 11:56:27.990447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:27.990082    2551 generic.go:356] "Generic (PLEG): container finished" podID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerID="a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9" exitCode=1
Jul 29 11:56:27.990447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:27.990158    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerDied","Data":"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"}
Jul 29 11:56:28.996530 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:28.996503    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/0.log"
Jul 29 11:56:28.996905 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:28.996578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerStarted","Data":"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"}
Jul 29 11:56:32.234283 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:32.234252    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:32.234672 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:32.234435    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer" containerID="cri-o://64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330" gracePeriod=30
Jul 29 11:56:32.885218 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:32.885197    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/1.log"
Jul 29 11:56:32.885612 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:32.885595    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/0.log"
Jul 29 11:56:32.887993 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:32.887976    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:33.014017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.013997    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/1.log"
Jul 29 11:56:33.014389 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014374    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg_fbd59370-bab5-4f03-8199-b30ee99671cb/storage-initializer/0.log"
Jul 29 11:56:33.014448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014408    2551 generic.go:356] "Generic (PLEG): container finished" podID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerID="64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330" exitCode=1
Jul 29 11:56:33.014448 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerDied","Data":"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"}
Jul 29 11:56:33.014524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014465    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"
Jul 29 11:56:33.014524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014492    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg" event={"ID":"fbd59370-bab5-4f03-8199-b30ee99671cb","Type":"ContainerDied","Data":"96225962b17d02e9e17124e85048428f10d367e1d80573e7a495b1ed7614c6ca"}
Jul 29 11:56:33.014524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.014521    2551 scope.go:117] "RemoveContainer" containerID="64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"
Jul 29 11:56:33.015274 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015260    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config\") pod \"fbd59370-bab5-4f03-8199-b30ee99671cb\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") "
Jul 29 11:56:33.015359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015304    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v87wd\" (UniqueName: \"kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd\") pod \"fbd59370-bab5-4f03-8199-b30ee99671cb\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") "
Jul 29 11:56:33.015359 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015330    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location\") pod \"fbd59370-bab5-4f03-8199-b30ee99671cb\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") "
Jul 29 11:56:33.015447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015365    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls\") pod \"fbd59370-bab5-4f03-8199-b30ee99671cb\" (UID: \"fbd59370-bab5-4f03-8199-b30ee99671cb\") "
Jul 29 11:56:33.015662 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015632    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbd59370-bab5-4f03-8199-b30ee99671cb" (UID: "fbd59370-bab5-4f03-8199-b30ee99671cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:33.015787 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.015661    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbd59370-bab5-4f03-8199-b30ee99671cb" (UID: "fbd59370-bab5-4f03-8199-b30ee99671cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:56:33.017553 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.017526    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbd59370-bab5-4f03-8199-b30ee99671cb" (UID: "fbd59370-bab5-4f03-8199-b30ee99671cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:56:33.017553 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.017542    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd" (OuterVolumeSpecName: "kube-api-access-v87wd") pod "fbd59370-bab5-4f03-8199-b30ee99671cb" (UID: "fbd59370-bab5-4f03-8199-b30ee99671cb"). InnerVolumeSpecName "kube-api-access-v87wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:56:33.030895 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.030877    2551 scope.go:117] "RemoveContainer" containerID="a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"
Jul 29 11:56:33.037816 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.037800    2551 scope.go:117] "RemoveContainer" containerID="64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"
Jul 29 11:56:33.038090 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:33.038073    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330\": container with ID starting with 64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330 not found: ID does not exist" containerID="64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"
Jul 29 11:56:33.038166 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.038097    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330"} err="failed to get container status \"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330\": rpc error: code = NotFound desc = could not find container \"64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330\": container with ID starting with 64f1a809d6598e36338e5dfbf4fedc0df9d7545c1d078ab4f17cc19037635330 not found: ID does not exist"
Jul 29 11:56:33.038166 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.038116    2551 scope.go:117] "RemoveContainer" containerID="a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"
Jul 29 11:56:33.038421 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:33.038400    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9\": container with ID starting with a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9 not found: ID does not exist" containerID="a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"
Jul 29 11:56:33.038503 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.038425    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9"} err="failed to get container status \"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9\": rpc error: code = NotFound desc = could not find container \"a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9\": container with ID starting with a218e1e0636c30cc78cf703c62cf8f4ebd4eb798979208d496d1021c298dd9b9 not found: ID does not exist"
Jul 29 11:56:33.116514 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.116495    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:33.116598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.116517    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v87wd\" (UniqueName: \"kubernetes.io/projected/fbd59370-bab5-4f03-8199-b30ee99671cb-kube-api-access-v87wd\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:33.116598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.116532    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbd59370-bab5-4f03-8199-b30ee99671cb-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:33.116598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.116543    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbd59370-bab5-4f03-8199-b30ee99671cb-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:56:33.340424 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340396    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:56:33.340708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340686    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.340708 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340697    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340709    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="storage-initializer"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340716    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="storage-initializer"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340725    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340731    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340738    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kube-rbac-proxy"
Jul 29 11:56:33.340783 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340743    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kube-rbac-proxy"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340814    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kserve-container"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340824    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340831    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e888f909-2248-428a-8f77-118a6ed96366" containerName="kube-rbac-proxy"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340838    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340889    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.340960 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.340894    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" containerName="storage-initializer"
Jul 29 11:56:33.345523 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.345504    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.347791 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.347772    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 29 11:56:33.347918 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.347897    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 29 11:56:33.347989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.347943    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 29 11:56:33.347989 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.347977    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:56:33.348330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.348312    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 29 11:56:33.348411 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.348353    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fbsmh\""
Jul 29 11:56:33.348463 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.348421    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 29 11:56:33.354524 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.354504    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:33.357316 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.357195    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:56:33.358971 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.358951    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-674d7b9598-gjzbg"]
Jul 29 11:56:33.519660 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.519633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.519760 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.519674    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zxt\" (UniqueName: \"kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.519830 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.519762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.519830 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.519798    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.520028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.519891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621278 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621289    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621311    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621469 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621337    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zxt\" (UniqueName: \"kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621469 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:33.621443    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 29 11:56:33.621747 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:56:33.621507    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls podName:2625666f-0b01-40f3-b7b4-f1e4d8a9b904 nodeName:}" failed. No retries permitted until 2026-07-29 11:56:34.12149023 +0000 UTC m=+5246.023718609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 29 11:56:33.621818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621768    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.621977 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.621954    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.622041 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.622018    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:33.632252 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:33.632227    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zxt\" (UniqueName: \"kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:34.125593 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:34.125564    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:34.128138 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:34.128111    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:34.257619 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:34.257588    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:34.385876 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:34.385851    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:56:34.388563 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:56:34.388537    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2625666f_0b01_40f3_b7b4_f1e4d8a9b904.slice/crio-055aa6d40907615aed0885f34b1041ab91db6afeb4dac1f565b6da4a4a27bb87 WatchSource:0}: Error finding container 055aa6d40907615aed0885f34b1041ab91db6afeb4dac1f565b6da4a4a27bb87: Status 404 returned error can't find the container with id 055aa6d40907615aed0885f34b1041ab91db6afeb4dac1f565b6da4a4a27bb87
Jul 29 11:56:34.728557 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:34.728530    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd59370-bab5-4f03-8199-b30ee99671cb" path="/var/lib/kubelet/pods/fbd59370-bab5-4f03-8199-b30ee99671cb/volumes"
Jul 29 11:56:35.026354 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:35.026273    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerStarted","Data":"6dc32f63064b1dd8fa022224e346bbb6ad84ebf7a4c272dfe285dca2ec21b0b7"}
Jul 29 11:56:35.026479 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:35.026366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerStarted","Data":"055aa6d40907615aed0885f34b1041ab91db6afeb4dac1f565b6da4a4a27bb87"}
Jul 29 11:56:36.037144 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:36.037099    2551 generic.go:356] "Generic (PLEG): container finished" podID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerID="6dc32f63064b1dd8fa022224e346bbb6ad84ebf7a4c272dfe285dca2ec21b0b7" exitCode=0
Jul 29 11:56:36.037596 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:36.037199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerDied","Data":"6dc32f63064b1dd8fa022224e346bbb6ad84ebf7a4c272dfe285dca2ec21b0b7"}
Jul 29 11:56:37.044275 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.044240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerStarted","Data":"b5f5c923a229a6f688c8388e2c51f2d85fa8805eeee12d1ca5eee0cf346779d7"}
Jul 29 11:56:37.044681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.044296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerStarted","Data":"134673eb5eae4736639bcf4fe4b18672568b832920a0d2825f55c1a47bd5918e"}
Jul 29 11:56:37.044681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.044514    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:37.044681 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.044557    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:37.045717 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.045688    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:56:37.065779 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:37.065742    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podStartSLOduration=3.057009749 podStartE2EDuration="4.065729425s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.008719676s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:56:37.063658176 +0000 UTC m=+5248.965886585" watchObservedRunningTime="2026-07-29 11:56:37.065729425 +0000 UTC m=+5248.967957822"
Jul 29 11:56:38.048909 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:38.048864    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:56:43.053868 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:43.053840    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:56:43.054353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:43.054327    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:56:53.054709 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:56:53.054671    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:03.055146 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:03.055094    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:13.054840 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:13.054805    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:23.054733 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:23.054658    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:33.054995 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:33.054953    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:43.054293 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:43.054257    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.78:8080: connect: connection refused"
Jul 29 11:57:53.055848 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:53.055811    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:57:53.372696 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:53.372608    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:57:53.383556 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:53.383512    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container" containerID="cri-o://134673eb5eae4736639bcf4fe4b18672568b832920a0d2825f55c1a47bd5918e" gracePeriod=30
Jul 29 11:57:53.383728 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:53.383530    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kube-rbac-proxy" containerID="cri-o://b5f5c923a229a6f688c8388e2c51f2d85fa8805eeee12d1ca5eee0cf346779d7" gracePeriod=30
Jul 29 11:57:54.391497 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.391441    2551 generic.go:356] "Generic (PLEG): container finished" podID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerID="b5f5c923a229a6f688c8388e2c51f2d85fa8805eeee12d1ca5eee0cf346779d7" exitCode=2
Jul 29 11:57:54.391868 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.391506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerDied","Data":"b5f5c923a229a6f688c8388e2c51f2d85fa8805eeee12d1ca5eee0cf346779d7"}
Jul 29 11:57:54.450581 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.450553    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:57:54.454092 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.454076    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.456498 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.456478    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 29 11:57:54.456602 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.456524    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 29 11:57:54.463931 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.463907    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:57:54.590487 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.590461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmrr\" (UniqueName: \"kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.590613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.590501    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.590613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.590528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.590729 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.590605    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691391 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmrr\" (UniqueName: \"kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691391 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691370    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691560 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691560 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691497    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691673 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:57:54.691638    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:57:54.691725 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:57:54.691711    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls podName:e661d8a5-43ed-41c0-ab73-8e496f3af2a5 nodeName:}" failed. No retries permitted until 2026-07-29 11:57:55.191692471 +0000 UTC m=+5327.093920868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:57:54.691781 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.691983 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.691966    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:54.699917 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:54.699897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmrr\" (UniqueName: \"kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:55.193809 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:55.193775    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:55.196186 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:55.196155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:55.368973 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:55.368942    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:57:55.501249 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:55.501221    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:57:56.401437 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:56.401408    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerStarted","Data":"a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb"}
Jul 29 11:57:56.401600 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:56.401459    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerStarted","Data":"ba16a234ef4683285a1fc6069a76dad861c84fb84ede51ba51fb781d27bc98b2"}
Jul 29 11:57:57.408309 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.408281    2551 generic.go:356] "Generic (PLEG): container finished" podID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerID="134673eb5eae4736639bcf4fe4b18672568b832920a0d2825f55c1a47bd5918e" exitCode=0
Jul 29 11:57:57.408625 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.408351    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerDied","Data":"134673eb5eae4736639bcf4fe4b18672568b832920a0d2825f55c1a47bd5918e"}
Jul 29 11:57:57.528313 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.528294    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:57:57.613053 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613001    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7zxt\" (UniqueName: \"kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt\") pod \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") "
Jul 29 11:57:57.613053 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613028    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location\") pod \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") "
Jul 29 11:57:57.613263 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613072    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config\") pod \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") "
Jul 29 11:57:57.613263 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613089    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert\") pod \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") "
Jul 29 11:57:57.613263 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613177    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") pod \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\" (UID: \"2625666f-0b01-40f3-b7b4-f1e4d8a9b904\") "
Jul 29 11:57:57.613441 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613374    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2625666f-0b01-40f3-b7b4-f1e4d8a9b904" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:57:57.613517 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613494    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2625666f-0b01-40f3-b7b4-f1e4d8a9b904" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:57:57.613583 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.613566    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2625666f-0b01-40f3-b7b4-f1e4d8a9b904" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:57:57.614985 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.614965    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt" (OuterVolumeSpecName: "kube-api-access-h7zxt") pod "2625666f-0b01-40f3-b7b4-f1e4d8a9b904" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904"). InnerVolumeSpecName "kube-api-access-h7zxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:57:57.615065 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.615028    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2625666f-0b01-40f3-b7b4-f1e4d8a9b904" (UID: "2625666f-0b01-40f3-b7b4-f1e4d8a9b904"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:57:57.714086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.714062    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:57:57.714086 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.714083    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zxt\" (UniqueName: \"kubernetes.io/projected/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-api-access-h7zxt\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:57:57.714265 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.714092    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:57:57.714265 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.714101    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:57:57.714265 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:57.714109    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2625666f-0b01-40f3-b7b4-f1e4d8a9b904-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:57:58.415149 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.415101    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"
Jul 29 11:57:58.415149 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.415114    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d" event={"ID":"2625666f-0b01-40f3-b7b4-f1e4d8a9b904","Type":"ContainerDied","Data":"055aa6d40907615aed0885f34b1041ab91db6afeb4dac1f565b6da4a4a27bb87"}
Jul 29 11:57:58.415707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.415214    2551 scope.go:117] "RemoveContainer" containerID="b5f5c923a229a6f688c8388e2c51f2d85fa8805eeee12d1ca5eee0cf346779d7"
Jul 29 11:57:58.424797 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.424768    2551 scope.go:117] "RemoveContainer" containerID="134673eb5eae4736639bcf4fe4b18672568b832920a0d2825f55c1a47bd5918e"
Jul 29 11:57:58.432864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.432846    2551 scope.go:117] "RemoveContainer" containerID="6dc32f63064b1dd8fa022224e346bbb6ad84ebf7a4c272dfe285dca2ec21b0b7"
Jul 29 11:57:58.438447 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.438419    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:57:58.442015 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.441996    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b96c9878-t545d"]
Jul 29 11:57:58.728961 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:57:58.728931    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" path="/var/lib/kubelet/pods/2625666f-0b01-40f3-b7b4-f1e4d8a9b904/volumes"
Jul 29 11:58:00.433846 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:00.433817    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/0.log"
Jul 29 11:58:00.434366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:00.433856    2551 generic.go:356] "Generic (PLEG): container finished" podID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerID="a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb" exitCode=1
Jul 29 11:58:00.434366 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:00.433897    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerDied","Data":"a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb"}
Jul 29 11:58:01.439697 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:01.439664    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/0.log"
Jul 29 11:58:01.440187 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:01.439729    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerStarted","Data":"5975a8411099cd6bcceb2b826bc4f45c7be2d960ea31a2fccdb5a7a3960137da"}
Jul 29 11:58:04.456898 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.456866    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/1.log"
Jul 29 11:58:04.457421 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.457402    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/0.log"
Jul 29 11:58:04.457470 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.457440    2551 generic.go:356] "Generic (PLEG): container finished" podID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerID="5975a8411099cd6bcceb2b826bc4f45c7be2d960ea31a2fccdb5a7a3960137da" exitCode=1
Jul 29 11:58:04.457507 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.457491    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerDied","Data":"5975a8411099cd6bcceb2b826bc4f45c7be2d960ea31a2fccdb5a7a3960137da"}
Jul 29 11:58:04.457561 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.457549    2551 scope.go:117] "RemoveContainer" containerID="a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb"
Jul 29 11:58:04.457946 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.457914    2551 scope.go:117] "RemoveContainer" containerID="a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb"
Jul 29 11:58:04.462628 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:04.462604    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:58:04.470729 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:04.470697    2551 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_kserve-ci-e2e-test_e661d8a5-43ed-41c0-ab73-8e496f3af2a5_0 in pod sandbox ba16a234ef4683285a1fc6069a76dad861c84fb84ede51ba51fb781d27bc98b2 from index: no such id: 'a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb'" containerID="a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb"
Jul 29 11:58:04.470838 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:04.470742    2551 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_kserve-ci-e2e-test_e661d8a5-43ed-41c0-ab73-8e496f3af2a5_0 in pod sandbox ba16a234ef4683285a1fc6069a76dad861c84fb84ede51ba51fb781d27bc98b2 from index: no such id: 'a46d66d14db356af4cfd78c1eea21d9ea533177faff847b2cec13164e88925cb'; Skipping pod \"isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_kserve-ci-e2e-test(e661d8a5-43ed-41c0-ab73-8e496f3af2a5)\"" logger="UnhandledError"
Jul 29 11:58:04.470957 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:04.470938    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_kserve-ci-e2e-test(e661d8a5-43ed-41c0-ab73-8e496f3af2a5)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5"
Jul 29 11:58:04.562467 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:04.562438    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:58:04.562594 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:04.562513    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls podName:e661d8a5-43ed-41c0-ab73-8e496f3af2a5 nodeName:}" failed. No retries permitted until 2026-07-29 11:58:05.062495276 +0000 UTC m=+5336.964723651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:58:05.066764 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:05.066731    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:58:05.066951 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:58:05.066808    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls podName:e661d8a5-43ed-41c0-ab73-8e496f3af2a5 nodeName:}" failed. No retries permitted until 2026-07-29 11:58:06.066792934 +0000 UTC m=+5337.969021309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 29 11:58:05.463510 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.463484    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/1.log"
Jul 29 11:58:05.567699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.567666    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:58:05.568090 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568072    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="storage-initializer"
Jul 29 11:58:05.568204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568092    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="storage-initializer"
Jul 29 11:58:05.568204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568119    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kube-rbac-proxy"
Jul 29 11:58:05.568204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568143    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kube-rbac-proxy"
Jul 29 11:58:05.568204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568159    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container"
Jul 29 11:58:05.568204 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568168    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container"
Jul 29 11:58:05.568472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568262    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kserve-container"
Jul 29 11:58:05.568472 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.568278    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2625666f-0b01-40f3-b7b4-f1e4d8a9b904" containerName="kube-rbac-proxy"
Jul 29 11:58:05.571453 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.571433    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.573895 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.573877    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:58:05.574005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.573906    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 29 11:58:05.574005 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.573877    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 29 11:58:05.583762 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.583638    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:58:05.597599 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.597582    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/1.log"
Jul 29 11:58:05.599829 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.599814    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:58:05.671560 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671536    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location\") pod \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") "
Jul 29 11:58:05.671646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkmrr\" (UniqueName: \"kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr\") pod \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") "
Jul 29 11:58:05.671699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671645    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config\") pod \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") "
Jul 29 11:58:05.671699 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671692    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") pod \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\" (UID: \"e661d8a5-43ed-41c0-ab73-8e496f3af2a5\") "
Jul 29 11:58:05.671805 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671774    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e661d8a5-43ed-41c0-ab73-8e496f3af2a5" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:58:05.671805 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671787    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.671929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671826    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.671929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671863    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnkh\" (UniqueName: \"kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.671929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671918    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.672075 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671941    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.672075 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.671979    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:58:05.672075 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.672051    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e661d8a5-43ed-41c0-ab73-8e496f3af2a5" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:58:05.673707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.673687    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr" (OuterVolumeSpecName: "kube-api-access-lkmrr") pod "e661d8a5-43ed-41c0-ab73-8e496f3af2a5" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5"). InnerVolumeSpecName "kube-api-access-lkmrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:58:05.673794 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.673769    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e661d8a5-43ed-41c0-ab73-8e496f3af2a5" (UID: "e661d8a5-43ed-41c0-ab73-8e496f3af2a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:58:05.772676 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772625    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnkh\" (UniqueName: \"kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.772676 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772725    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772770    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmrr\" (UniqueName: \"kubernetes.io/projected/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-api-access-lkmrr\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772781    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:58:05.772807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.772791    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e661d8a5-43ed-41c0-ab73-8e496f3af2a5-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:58:05.773186 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.773170    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.773439 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.773416    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.773527 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.773418    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.775003 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.774983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.780056 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.780032    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnkh\" (UniqueName: \"kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:05.884549 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:05.884516    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:06.009929 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.009904    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:58:06.011687 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:58:06.011662    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8e0438_0cdf_45a0_abc6_e424ec663716.slice/crio-1b67ac8558b92d9292054a03f1ecd3faeb15be6c1e4f9d4e8992ce8136894a75 WatchSource:0}: Error finding container 1b67ac8558b92d9292054a03f1ecd3faeb15be6c1e4f9d4e8992ce8136894a75: Status 404 returned error can't find the container with id 1b67ac8558b92d9292054a03f1ecd3faeb15be6c1e4f9d4e8992ce8136894a75
Jul 29 11:58:06.469185 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.469164    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv_e661d8a5-43ed-41c0-ab73-8e496f3af2a5/storage-initializer/1.log"
Jul 29 11:58:06.469599 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.469265    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"
Jul 29 11:58:06.469599 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.469270    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv" event={"ID":"e661d8a5-43ed-41c0-ab73-8e496f3af2a5","Type":"ContainerDied","Data":"ba16a234ef4683285a1fc6069a76dad861c84fb84ede51ba51fb781d27bc98b2"}
Jul 29 11:58:06.469599 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.469341    2551 scope.go:117] "RemoveContainer" containerID="5975a8411099cd6bcceb2b826bc4f45c7be2d960ea31a2fccdb5a7a3960137da"
Jul 29 11:58:06.471260 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.471232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerStarted","Data":"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"}
Jul 29 11:58:06.471363 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.471317    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerStarted","Data":"1b67ac8558b92d9292054a03f1ecd3faeb15be6c1e4f9d4e8992ce8136894a75"}
Jul 29 11:58:06.517564 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.517539    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:58:06.522480 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.522457    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-55d588cc5d-ldzrv"]
Jul 29 11:58:06.728580 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:06.728511    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" path="/var/lib/kubelet/pods/e661d8a5-43ed-41c0-ab73-8e496f3af2a5/volumes"
Jul 29 11:58:07.476725 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:07.476691    2551 generic.go:356] "Generic (PLEG): container finished" podID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerID="588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444" exitCode=0
Jul 29 11:58:07.477102 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:07.476779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerDied","Data":"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"}
Jul 29 11:58:08.484378 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:08.484344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerStarted","Data":"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"}
Jul 29 11:58:08.484782 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:08.484405    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerStarted","Data":"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"}
Jul 29 11:58:08.484782 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:08.484433    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:08.504722 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:08.504678    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podStartSLOduration=2.5041353170000002 podStartE2EDuration="3.504662401s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.000527084s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:58:08.502402621 +0000 UTC m=+5340.404631019" watchObservedRunningTime="2026-07-29 11:58:08.504662401 +0000 UTC m=+5340.406890798"
Jul 29 11:58:09.489312 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:09.489281    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:09.490350 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:09.490321    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:10.493887 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:10.493846    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:15.499048 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:15.499015    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:58:15.499664 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:15.499634    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:25.500300 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:25.500258    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:35.499715 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:35.499668    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:45.499706 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:45.499664    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:58:55.500406 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:58:55.500360    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:59:05.499660 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:05.499572    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:59:09.045422 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:09.045395    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:59:09.056451 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:09.056430    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 11:59:15.500025 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:15.499979    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:59:18.728012 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:18.727976    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:59:25.603807 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:25.603771    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:59:25.604228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:25.604038    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" containerID="cri-o://5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff" gracePeriod=30
Jul 29 11:59:25.604228 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:25.604142    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kube-rbac-proxy" containerID="cri-o://33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036" gracePeriod=30
Jul 29 11:59:25.804986 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:25.804955    2551 generic.go:356] "Generic (PLEG): container finished" podID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerID="33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036" exitCode=2
Jul 29 11:59:25.805161 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:25.805025    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerDied","Data":"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"}
Jul 29 11:59:26.674170 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674118    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:26.674652 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674634    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.674744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674656    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.674744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674672    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.674744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674680    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.674905 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674794    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.674971 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.674957    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e661d8a5-43ed-41c0-ab73-8e496f3af2a5" containerName="storage-initializer"
Jul 29 11:59:26.677994 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.677975    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.680330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.680308    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 29 11:59:26.680438 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.680350    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 29 11:59:26.688613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.688591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:26.748198 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.748173    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.748286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.748212    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.748286 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.748234    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdt2\" (UniqueName: \"kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.748364 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.748286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.849613 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.849586    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.849720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.849623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.849720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.849645    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdt2\" (UniqueName: \"kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.849720 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.849668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.850052 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.850035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.850342 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.850318    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.852104 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.852085    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.856829 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.856806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdt2\" (UniqueName: \"kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:26.991030 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:26.991009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:27.118326 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:27.118298    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:27.120508 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:59:27.120483    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ebb6cd_1a69_444d_b6f9_b3745623691f.slice/crio-9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6 WatchSource:0}: Error finding container 9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6: Status 404 returned error can't find the container with id 9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6
Jul 29 11:59:27.815535 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:27.815497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerStarted","Data":"ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"}
Jul 29 11:59:27.815928 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:27.815564    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerStarted","Data":"9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6"}
Jul 29 11:59:28.725107 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:28.725067    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.80:8080: connect: connection refused"
Jul 29 11:59:29.705089 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.705067    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:59:29.774103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert\") pod \"0d8e0438-0cdf-45a0-abc6-e424ec663716\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") "
Jul 29 11:59:29.774103 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774087    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location\") pod \"0d8e0438-0cdf-45a0-abc6-e424ec663716\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") "
Jul 29 11:59:29.774246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774194    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls\") pod \"0d8e0438-0cdf-45a0-abc6-e424ec663716\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") "
Jul 29 11:59:29.774246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774218    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hnkh\" (UniqueName: \"kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh\") pod \"0d8e0438-0cdf-45a0-abc6-e424ec663716\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") "
Jul 29 11:59:29.774246 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774236    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config\") pod \"0d8e0438-0cdf-45a0-abc6-e424ec663716\" (UID: \"0d8e0438-0cdf-45a0-abc6-e424ec663716\") "
Jul 29 11:59:29.774456 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774431    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d8e0438-0cdf-45a0-abc6-e424ec663716" (UID: "0d8e0438-0cdf-45a0-abc6-e424ec663716"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:59:29.774530 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774451    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d8e0438-0cdf-45a0-abc6-e424ec663716" (UID: "0d8e0438-0cdf-45a0-abc6-e424ec663716"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:29.774679 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.774657    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d8e0438-0cdf-45a0-abc6-e424ec663716" (UID: "0d8e0438-0cdf-45a0-abc6-e424ec663716"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:59:29.776258 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.776237    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh" (OuterVolumeSpecName: "kube-api-access-7hnkh") pod "0d8e0438-0cdf-45a0-abc6-e424ec663716" (UID: "0d8e0438-0cdf-45a0-abc6-e424ec663716"). InnerVolumeSpecName "kube-api-access-7hnkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:59:29.776385 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.776372    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d8e0438-0cdf-45a0-abc6-e424ec663716" (UID: "0d8e0438-0cdf-45a0-abc6-e424ec663716"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:59:29.824900 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.824881    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/0.log"
Jul 29 11:59:29.824980 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.824913    2551 generic.go:356] "Generic (PLEG): container finished" podID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerID="ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96" exitCode=1
Jul 29 11:59:29.825028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.824992    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerDied","Data":"ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"}
Jul 29 11:59:29.832894 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.832870    2551 generic.go:356] "Generic (PLEG): container finished" podID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerID="5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff" exitCode=0
Jul 29 11:59:29.833017 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.832980    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerDied","Data":"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"}
Jul 29 11:59:29.833111 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.833092    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"
Jul 29 11:59:29.833111 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.833100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr" event={"ID":"0d8e0438-0cdf-45a0-abc6-e424ec663716","Type":"ContainerDied","Data":"1b67ac8558b92d9292054a03f1ecd3faeb15be6c1e4f9d4e8992ce8136894a75"}
Jul 29 11:59:29.833919 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.833171    2551 scope.go:117] "RemoveContainer" containerID="33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"
Jul 29 11:59:29.842567 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.842548    2551 scope.go:117] "RemoveContainer" containerID="5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"
Jul 29 11:59:29.852536 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.852518    2551 scope.go:117] "RemoveContainer" containerID="588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"
Jul 29 11:59:29.860026 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.860003    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:59:29.863215 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.863195    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5448c7f489-w65zr"]
Jul 29 11:59:29.867051 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867019    2551 scope.go:117] "RemoveContainer" containerID="33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"
Jul 29 11:59:29.867403 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:59:29.867383    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036\": container with ID starting with 33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036 not found: ID does not exist" containerID="33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"
Jul 29 11:59:29.867468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867409    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036"} err="failed to get container status \"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036\": rpc error: code = NotFound desc = could not find container \"33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036\": container with ID starting with 33a1ee91a768f45dc403091c8d60eabd19dfc072a038c6a06b5bc90412ff0036 not found: ID does not exist"
Jul 29 11:59:29.867468 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867426    2551 scope.go:117] "RemoveContainer" containerID="5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"
Jul 29 11:59:29.867706 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:59:29.867690    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff\": container with ID starting with 5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff not found: ID does not exist" containerID="5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"
Jul 29 11:59:29.867752 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867712    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff"} err="failed to get container status \"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff\": rpc error: code = NotFound desc = could not find container \"5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff\": container with ID starting with 5bc0688363cf55ff30eb738797c423551ab6b29c10cf5093303fad389278baff not found: ID does not exist"
Jul 29 11:59:29.867752 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867736    2551 scope.go:117] "RemoveContainer" containerID="588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"
Jul 29 11:59:29.867986 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:59:29.867973    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444\": container with ID starting with 588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444 not found: ID does not exist" containerID="588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"
Jul 29 11:59:29.868028 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.867988    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444"} err="failed to get container status \"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444\": rpc error: code = NotFound desc = could not find container \"588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444\": container with ID starting with 588888f5a113ddac1f234fc2deb4bc0bd9e2b341f1e6ad75ca658ee7cfad7444 not found: ID does not exist"
Jul 29 11:59:29.874864 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.874845    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e0438-0cdf-45a0-abc6-e424ec663716-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:29.874949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.874868    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnkh\" (UniqueName: \"kubernetes.io/projected/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-api-access-7hnkh\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:29.874949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.874882    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:29.874949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.874897    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d8e0438-0cdf-45a0-abc6-e424ec663716-cabundle-cert\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:29.874949 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:29.874909    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d8e0438-0cdf-45a0-abc6-e424ec663716-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:30.728335 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:30.728303    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" path="/var/lib/kubelet/pods/0d8e0438-0cdf-45a0-abc6-e424ec663716/volumes"
Jul 29 11:59:30.838817 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:30.838788    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/0.log"
Jul 29 11:59:30.838990 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:30.838902    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerStarted","Data":"535153537b450b9ec88aa813d17637e99a373deb4e6733341cedf1f59a8fba3c"}
Jul 29 11:59:35.864906 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.864879    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/1.log"
Jul 29 11:59:35.865363 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.865348    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/0.log"
Jul 29 11:59:35.865431 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.865380    2551 generic.go:356] "Generic (PLEG): container finished" podID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerID="535153537b450b9ec88aa813d17637e99a373deb4e6733341cedf1f59a8fba3c" exitCode=1
Jul 29 11:59:35.865488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.865460    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerDied","Data":"535153537b450b9ec88aa813d17637e99a373deb4e6733341cedf1f59a8fba3c"}
Jul 29 11:59:35.865545 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.865533    2551 scope.go:117] "RemoveContainer" containerID="ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"
Jul 29 11:59:35.865800 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.865785    2551 scope.go:117] "RemoveContainer" containerID="ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"
Jul 29 11:59:35.876385 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:59:35.876356    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_kserve-ci-e2e-test(a6ebb6cd-1a69-444d-b6f9-b3745623691f)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f"
Jul 29 11:59:35.876524 ip-10-0-134-91 kubenswrapper[2551]: E0729 11:59:35.876498    2551 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_kserve-ci-e2e-test_a6ebb6cd-1a69-444d-b6f9-b3745623691f_0 in pod sandbox 9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6 from index: no such id: 'ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96'" containerID="ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"
Jul 29 11:59:35.876598 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:35.876547    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_kserve-ci-e2e-test_a6ebb6cd-1a69-444d-b6f9-b3745623691f_0 in pod sandbox 9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6 from index: no such id: 'ba0b1d29106de6576dbe4a755272af6bade4e29345699f8a9cde8ca771d4aa96'"
Jul 29 11:59:36.676950 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:36.676916    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:36.871859 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:36.871824    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/1.log"
Jul 29 11:59:36.999323 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:36.999301    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/1.log"
Jul 29 11:59:37.001353 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.001337    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:37.130646 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.130623    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwdt2\" (UniqueName: \"kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2\") pod \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") "
Jul 29 11:59:37.130744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.130663    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location\") pod \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") "
Jul 29 11:59:37.130744 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.130738    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls\") pod \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") "
Jul 29 11:59:37.130818 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.130770    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config\") pod \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\" (UID: \"a6ebb6cd-1a69-444d-b6f9-b3745623691f\") "
Jul 29 11:59:37.131047 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.130989    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6ebb6cd-1a69-444d-b6f9-b3745623691f" (UID: "a6ebb6cd-1a69-444d-b6f9-b3745623691f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:37.131182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.131151    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6ebb6cd-1a69-444d-b6f9-b3745623691f" (UID: "a6ebb6cd-1a69-444d-b6f9-b3745623691f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:59:37.132707 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.132675    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2" (OuterVolumeSpecName: "kube-api-access-xwdt2") pod "a6ebb6cd-1a69-444d-b6f9-b3745623691f" (UID: "a6ebb6cd-1a69-444d-b6f9-b3745623691f"). InnerVolumeSpecName "kube-api-access-xwdt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:59:37.132786 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.132720    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6ebb6cd-1a69-444d-b6f9-b3745623691f" (UID: "a6ebb6cd-1a69-444d-b6f9-b3745623691f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:59:37.232281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.232260    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6ebb6cd-1a69-444d-b6f9-b3745623691f-proxy-tls\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:37.232281 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.232280    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:37.232401 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.232289    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwdt2\" (UniqueName: \"kubernetes.io/projected/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kube-api-access-xwdt2\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:37.232401 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.232298    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6ebb6cd-1a69-444d-b6f9-b3745623691f-kserve-provision-location\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 11:59:37.878375 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.878342    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5_a6ebb6cd-1a69-444d-b6f9-b3745623691f/storage-initializer/1.log"
Jul 29 11:59:37.878773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.878459    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5" event={"ID":"a6ebb6cd-1a69-444d-b6f9-b3745623691f","Type":"ContainerDied","Data":"9b88297cfcada45a2e5c6823bb232d96f1fcd18857ff176e757b2c9b1a3ebcc6"}
Jul 29 11:59:37.878773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.878531    2551 scope.go:117] "RemoveContainer" containerID="535153537b450b9ec88aa813d17637e99a373deb4e6733341cedf1f59a8fba3c"
Jul 29 11:59:37.878773 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.878466    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"
Jul 29 11:59:37.914182 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.914155    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:37.917330 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:37.917303    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7547d7674c-l6zh5"]
Jul 29 11:59:38.728016 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.727970    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" path="/var/lib/kubelet/pods/a6ebb6cd-1a69-444d-b6f9-b3745623691f/volumes"
Jul 29 11:59:38.730079 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730054    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-849dz/must-gather-cpvb6"]
Jul 29 11:59:38.730503 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730487    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730507    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730527    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container"
Jul 29 11:59:38.730550 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730536    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730555    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="storage-initializer"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730564    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="storage-initializer"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730574    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kube-rbac-proxy"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730582    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kube-rbac-proxy"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730597    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730645 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730606    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730677    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730687    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kube-rbac-proxy"
Jul 29 11:59:38.730826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730698    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ebb6cd-1a69-444d-b6f9-b3745623691f" containerName="storage-initializer"
Jul 29 11:59:38.730826 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.730707    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8e0438-0cdf-45a0-abc6-e424ec663716" containerName="kserve-container"
Jul 29 11:59:38.735118 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.735098    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.737588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.737498    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-849dz\"/\"openshift-service-ca.crt\""
Jul 29 11:59:38.737588 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.737532    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-849dz\"/\"kube-root-ca.crt\""
Jul 29 11:59:38.740460 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.740434    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-849dz/must-gather-cpvb6"]
Jul 29 11:59:38.843388 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.843367    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9h47\" (UniqueName: \"kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.843493 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.843397    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.944188 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.944165    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9h47\" (UniqueName: \"kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.944488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.944192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.944488 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.944478    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:38.953865 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:38.953840    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9h47\" (UniqueName: \"kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47\") pod \"must-gather-cpvb6\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") " pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:39.052899 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:39.052845    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 11:59:39.180266 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:39.180240    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-849dz/must-gather-cpvb6"]
Jul 29 11:59:39.182191 ip-10-0-134-91 kubenswrapper[2551]: W0729 11:59:39.182159    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02fb6e9_d4f0_40e9_afca_f78be27690ce.slice/crio-32104e07f65640ff4e5abb1c94120e568dc2164f0812fa98bc6c8e9adfac879e WatchSource:0}: Error finding container 32104e07f65640ff4e5abb1c94120e568dc2164f0812fa98bc6c8e9adfac879e: Status 404 returned error can't find the container with id 32104e07f65640ff4e5abb1c94120e568dc2164f0812fa98bc6c8e9adfac879e
Jul 29 11:59:39.183703 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:39.183687    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 11:59:39.891402 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:39.891366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-849dz/must-gather-cpvb6" event={"ID":"d02fb6e9-d4f0-40e9-afca-f78be27690ce","Type":"ContainerStarted","Data":"32104e07f65640ff4e5abb1c94120e568dc2164f0812fa98bc6c8e9adfac879e"}
Jul 29 11:59:45.928436 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:45.928401    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-849dz/must-gather-cpvb6" event={"ID":"d02fb6e9-d4f0-40e9-afca-f78be27690ce","Type":"ContainerStarted","Data":"2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d"}
Jul 29 11:59:45.928823 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:45.928452    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-849dz/must-gather-cpvb6" event={"ID":"d02fb6e9-d4f0-40e9-afca-f78be27690ce","Type":"ContainerStarted","Data":"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"}
Jul 29 11:59:45.944466 ip-10-0-134-91 kubenswrapper[2551]: I0729 11:59:45.944423    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-849dz/must-gather-cpvb6" podStartSLOduration=1.873863546 podStartE2EDuration="7.944407879s" totalImagesPullingTime="6.070544333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:59:45.943423204 +0000 UTC m=+5437.845651602" watchObservedRunningTime="2026-07-29 11:59:45.944407879 +0000 UTC m=+5437.846636275"
Jul 29 12:00:12.065590 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.065556    2551 generic.go:356] "Generic (PLEG): container finished" podID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerID="674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615" exitCode=0
Jul 29 12:00:12.066013 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.065628    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-849dz/must-gather-cpvb6" event={"ID":"d02fb6e9-d4f0-40e9-afca-f78be27690ce","Type":"ContainerDied","Data":"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"}
Jul 29 12:00:12.066013 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.065946    2551 scope.go:117] "RemoveContainer" containerID="674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"
Jul 29 12:00:12.092989 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.092964    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-849dz_must-gather-cpvb6_d02fb6e9-d4f0-40e9-afca-f78be27690ce/gather/0.log"
Jul 29 12:00:12.767106 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.767074    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjdcv/must-gather-4n9xj"]
Jul 29 12:00:12.769411 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.769395    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.771848 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.771827    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjdcv\"/\"openshift-service-ca.crt\""
Jul 29 12:00:12.772843 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.772818    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjdcv\"/\"kube-root-ca.crt\""
Jul 29 12:00:12.772958 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.772861    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fjdcv\"/\"default-dockercfg-rh64t\""
Jul 29 12:00:12.778016 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.777992    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjdcv/must-gather-4n9xj"]
Jul 29 12:00:12.844377 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.844347    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b72722a-4d9f-4092-9e85-6b8d1e15850a-must-gather-output\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.844527 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.844394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/5b72722a-4d9f-4092-9e85-6b8d1e15850a-kube-api-access-hzqgd\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.945205 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.945167    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b72722a-4d9f-4092-9e85-6b8d1e15850a-must-gather-output\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.945382 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.945238    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/5b72722a-4d9f-4092-9e85-6b8d1e15850a-kube-api-access-hzqgd\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.945536 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.945512    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5b72722a-4d9f-4092-9e85-6b8d1e15850a-must-gather-output\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:12.953337 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:12.953311    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqgd\" (UniqueName: \"kubernetes.io/projected/5b72722a-4d9f-4092-9e85-6b8d1e15850a-kube-api-access-hzqgd\") pod \"must-gather-4n9xj\" (UID: \"5b72722a-4d9f-4092-9e85-6b8d1e15850a\") " pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:13.078790 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:13.078702    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjdcv/must-gather-4n9xj"
Jul 29 12:00:13.204640 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:13.204611    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjdcv/must-gather-4n9xj"]
Jul 29 12:00:13.207029 ip-10-0-134-91 kubenswrapper[2551]: W0729 12:00:13.206998    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b72722a_4d9f_4092_9e85_6b8d1e15850a.slice/crio-40a863e06b901dfc65482da8154fa995e51e71683d8d35b942f5577611975bbe WatchSource:0}: Error finding container 40a863e06b901dfc65482da8154fa995e51e71683d8d35b942f5577611975bbe: Status 404 returned error can't find the container with id 40a863e06b901dfc65482da8154fa995e51e71683d8d35b942f5577611975bbe
Jul 29 12:00:14.084423 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:14.084381    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjdcv/must-gather-4n9xj" event={"ID":"5b72722a-4d9f-4092-9e85-6b8d1e15850a","Type":"ContainerStarted","Data":"40a863e06b901dfc65482da8154fa995e51e71683d8d35b942f5577611975bbe"}
Jul 29 12:00:15.091591 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.091543    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjdcv/must-gather-4n9xj" event={"ID":"5b72722a-4d9f-4092-9e85-6b8d1e15850a","Type":"ContainerStarted","Data":"a0e60644b76490bd48a55eca7f3e18266447cdc1f5f7cda9224e97177f03936f"}
Jul 29 12:00:15.092108 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.091603    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjdcv/must-gather-4n9xj" event={"ID":"5b72722a-4d9f-4092-9e85-6b8d1e15850a","Type":"ContainerStarted","Data":"b0548f2084aa24f4414e86404bb7a531ce01bf573eda27a6cf2d8ac45210c7b4"}
Jul 29 12:00:15.109176 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.109111    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fjdcv/must-gather-4n9xj" podStartSLOduration=2.163541516 podStartE2EDuration="3.109093445s" totalImagesPullingTime="945.551929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:00:15.107175632 +0000 UTC m=+5467.009404029" watchObservedRunningTime="2026-07-29 12:00:15.109093445 +0000 UTC m=+5467.011321843"
Jul 29 12:00:15.682651 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.682624    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p5v7t_299a9e33-f924-4aea-b3b0-15bde11145dc/global-pull-secret-syncer/0.log"
Jul 29 12:00:15.832845 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.832819    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-979pr_a460fef0-8b06-44b2-b61c-1269fbb825db/konnectivity-agent/0.log"
Jul 29 12:00:15.908926 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:15.908896    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-91.ec2.internal_e25e3ac4c36569f4c5a0d44ba195e649/haproxy/0.log"
Jul 29 12:00:17.619604 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:17.619566    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-849dz/must-gather-cpvb6"]
Jul 29 12:00:17.623423 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:17.622556    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cpvb6\" is forbidden: User \"system:node:ip-10-0-134-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-849dz\": no relationship found between node 'ip-10-0-134-91.ec2.internal' and this object" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:17.624740 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:17.624719    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-849dz/must-gather-cpvb6"]
Jul 29 12:00:17.624925 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:17.624902    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-849dz/must-gather-cpvb6" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="copy" containerID="cri-o://2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d" gracePeriod=2
Jul 29 12:00:17.984003 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:17.983907    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-849dz_must-gather-cpvb6_d02fb6e9-d4f0-40e9-afca-f78be27690ce/copy/0.log"
Jul 29 12:00:17.993427 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:17.993094    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:17.995395 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:17.995344    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cpvb6\" is forbidden: User \"system:node:ip-10-0-134-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-849dz\": no relationship found between node 'ip-10-0-134-91.ec2.internal' and this object" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:18.100663 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.100622    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output\") pod \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") "
Jul 29 12:00:18.100982 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.100762    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9h47\" (UniqueName: \"kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47\") pod \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\" (UID: \"d02fb6e9-d4f0-40e9-afca-f78be27690ce\") "
Jul 29 12:00:18.102672 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.102638    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d02fb6e9-d4f0-40e9-afca-f78be27690ce" (UID: "d02fb6e9-d4f0-40e9-afca-f78be27690ce"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:00:18.111428 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.104980    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47" (OuterVolumeSpecName: "kube-api-access-d9h47") pod "d02fb6e9-d4f0-40e9-afca-f78be27690ce" (UID: "d02fb6e9-d4f0-40e9-afca-f78be27690ce"). InnerVolumeSpecName "kube-api-access-d9h47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:00:18.111428 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.111080    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-849dz_must-gather-cpvb6_d02fb6e9-d4f0-40e9-afca-f78be27690ce/copy/0.log"
Jul 29 12:00:18.111788 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.111483    2551 generic.go:356] "Generic (PLEG): container finished" podID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerID="2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d" exitCode=143
Jul 29 12:00:18.111788 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.111592    2551 scope.go:117] "RemoveContainer" containerID="2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d"
Jul 29 12:00:18.111788 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.111746    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:18.129459 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:18.128497    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cpvb6\" is forbidden: User \"system:node:ip-10-0-134-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-849dz\": no relationship found between node 'ip-10-0-134-91.ec2.internal' and this object" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:18.144169 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:18.141742    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cpvb6\" is forbidden: User \"system:node:ip-10-0-134-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-849dz\": no relationship found between node 'ip-10-0-134-91.ec2.internal' and this object" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:18.144169 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.142616    2551 scope.go:117] "RemoveContainer" containerID="674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"
Jul 29 12:00:18.175098 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.175069    2551 scope.go:117] "RemoveContainer" containerID="2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d"
Jul 29 12:00:18.175492 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:18.175466    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d\": container with ID starting with 2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d not found: ID does not exist" containerID="2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d"
Jul 29 12:00:18.175581 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.175505    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d"} err="failed to get container status \"2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d\": rpc error: code = NotFound desc = could not find container \"2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d\": container with ID starting with 2686634a2e57e5aeb422bacabb2076c29af2c2877d0b6efc5f0bb33a24411b0d not found: ID does not exist"
Jul 29 12:00:18.175581 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.175543    2551 scope.go:117] "RemoveContainer" containerID="674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"
Jul 29 12:00:18.176156 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:18.175803    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615\": container with ID starting with 674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615 not found: ID does not exist" containerID="674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"
Jul 29 12:00:18.176156 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.175851    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615"} err="failed to get container status \"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615\": rpc error: code = NotFound desc = could not find container \"674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615\": container with ID starting with 674cfa755754f398744ab5846e4ff2206eeb251e91e7a3ad9940701d77314615 not found: ID does not exist"
Jul 29 12:00:18.204150 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.202353    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9h47\" (UniqueName: \"kubernetes.io/projected/d02fb6e9-d4f0-40e9-afca-f78be27690ce-kube-api-access-d9h47\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 12:00:18.204150 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.202398    2551 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d02fb6e9-d4f0-40e9-afca-f78be27690ce-must-gather-output\") on node \"ip-10-0-134-91.ec2.internal\" DevicePath \"\""
Jul 29 12:00:18.730155 ip-10-0-134-91 kubenswrapper[2551]: E0729 12:00:18.730093    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-cpvb6\" is forbidden: User \"system:node:ip-10-0-134-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-849dz\": no relationship found between node 'ip-10-0-134-91.ec2.internal' and this object" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" pod="openshift-must-gather-849dz/must-gather-cpvb6"
Jul 29 12:00:18.735286 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:18.735163    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" path="/var/lib/kubelet/pods/d02fb6e9-d4f0-40e9-afca-f78be27690ce/volumes"
Jul 29 12:00:19.994759 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:19.994712    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/alertmanager/0.log"
Jul 29 12:00:20.052256 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.052223    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/config-reloader/0.log"
Jul 29 12:00:20.080466 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.080429    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/kube-rbac-proxy-web/0.log"
Jul 29 12:00:20.105150 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.104959    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/kube-rbac-proxy/0.log"
Jul 29 12:00:20.131150 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.129930    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/kube-rbac-proxy-metric/0.log"
Jul 29 12:00:20.152365 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.152334    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/prom-label-proxy/0.log"
Jul 29 12:00:20.180853 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.180804    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8104c6a7-e067-4125-af70-f9373ba7a66b/init-config-reloader/0.log"
Jul 29 12:00:20.336939 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.336871    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54dcf68874-2bvlv_7dde4a0c-1abb-417e-87b0-6d74828ac094/metrics-server/0.log"
Jul 29 12:00:20.362956 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.362853    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-6tb7s_af603e8f-d435-4811-a831-6fd9f980db74/monitoring-plugin/0.log"
Jul 29 12:00:20.397598 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.397566    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75wlb_07e5545f-be3d-4f06-9658-dbfaee2b7493/node-exporter/0.log"
Jul 29 12:00:20.418252 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.418192    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75wlb_07e5545f-be3d-4f06-9658-dbfaee2b7493/kube-rbac-proxy/0.log"
Jul 29 12:00:20.437945 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.437910    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75wlb_07e5545f-be3d-4f06-9658-dbfaee2b7493/init-textfile/0.log"
Jul 29 12:00:20.621840 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.621752    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-fz27k_f64cd852-e9fa-49be-8bb0-96531811ca1e/kube-rbac-proxy-main/0.log"
Jul 29 12:00:20.644252 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.644222    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-fz27k_f64cd852-e9fa-49be-8bb0-96531811ca1e/kube-rbac-proxy-self/0.log"
Jul 29 12:00:20.663073 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.663022    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-fz27k_f64cd852-e9fa-49be-8bb0-96531811ca1e/openshift-state-metrics/0.log"
Jul 29 12:00:20.715207 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.715168    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/prometheus/0.log"
Jul 29 12:00:20.731675 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.731649    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/config-reloader/0.log"
Jul 29 12:00:20.756966 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.756940    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/thanos-sidecar/0.log"
Jul 29 12:00:20.775460 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.775416    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/kube-rbac-proxy-web/0.log"
Jul 29 12:00:20.794618 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.794589    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/kube-rbac-proxy/0.log"
Jul 29 12:00:20.813297 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.813266    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/kube-rbac-proxy-thanos/0.log"
Jul 29 12:00:20.833569 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.833536    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4f0c4a1c-c0c1-401a-b311-95174b47250b/init-config-reloader/0.log"
Jul 29 12:00:20.912535 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:20.912447    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9248b_b09bf49f-251c-4e50-9a24-6e711b73e16f/prometheus-operator-admission-webhook/0.log"
Jul 29 12:00:21.014367 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.014330    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/thanos-query/0.log"
Jul 29 12:00:21.034534 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.034507    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/kube-rbac-proxy-web/0.log"
Jul 29 12:00:21.057646 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.057622    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/kube-rbac-proxy/0.log"
Jul 29 12:00:21.077811 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.077786    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/prom-label-proxy/0.log"
Jul 29 12:00:21.097141 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.097095    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/kube-rbac-proxy-rules/0.log"
Jul 29 12:00:21.117778 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:21.117754    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-748f5c9894-8fk2t_5f6b04c1-17f5-4dd3-93e7-483db67c5fcb/kube-rbac-proxy-metrics/0.log"
Jul 29 12:00:24.293846 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.293797    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"]
Jul 29 12:00:24.294962 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.294942    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="gather"
Jul 29 12:00:24.295069 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.295028    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="gather"
Jul 29 12:00:24.295069 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.295052    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="copy"
Jul 29 12:00:24.295069 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.295060    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="copy"
Jul 29 12:00:24.295258 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.295202    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="gather"
Jul 29 12:00:24.295258 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.295218    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02fb6e9-d4f0-40e9-afca-f78be27690ce" containerName="copy"
Jul 29 12:00:24.301293 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.301096    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.305603 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.304950    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"]
Jul 29 12:00:24.460648 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.460348    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-sys\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.460648 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.460434    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-lib-modules\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.460648 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.460465    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-podres\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.460648 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.460524    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-proc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.460648 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.460568    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbdc\" (UniqueName: \"kubernetes.io/projected/27360541-e32e-4218-8151-ec26a51f72ef-kube-api-access-lvbdc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.561967 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.561874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-lib-modules\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.561967 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.561920    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-podres\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.561984    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-proc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbdc\" (UniqueName: \"kubernetes.io/projected/27360541-e32e-4218-8151-ec26a51f72ef-kube-api-access-lvbdc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562050    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-lib-modules\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-sys\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-podres\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562153    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-proc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.562287 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.562160    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27360541-e32e-4218-8151-ec26a51f72ef-sys\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.570368 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.570343    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbdc\" (UniqueName: \"kubernetes.io/projected/27360541-e32e-4218-8151-ec26a51f72ef-kube-api-access-lvbdc\") pod \"perf-node-gather-daemonset-pdsjk\" (UID: \"27360541-e32e-4218-8151-ec26a51f72ef\") " pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.616344 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.616282    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:24.788793 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.788580    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"]
Jul 29 12:00:24.791561 ip-10-0-134-91 kubenswrapper[2551]: W0729 12:00:24.791533    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27360541_e32e_4218_8151_ec26a51f72ef.slice/crio-76bd3dd40edab7db4ce8a53bfdff40585423317ce6928a699767996bab5a037b WatchSource:0}: Error finding container 76bd3dd40edab7db4ce8a53bfdff40585423317ce6928a699767996bab5a037b: Status 404 returned error can't find the container with id 76bd3dd40edab7db4ce8a53bfdff40585423317ce6928a699767996bab5a037b
Jul 29 12:00:24.811504 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.811483    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-49kqc_80612ed9-585a-4273-9330-304fada0fe36/dns/0.log"
Jul 29 12:00:24.829849 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.829797    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-49kqc_80612ed9-585a-4273-9330-304fada0fe36/kube-rbac-proxy/0.log"
Jul 29 12:00:24.944082 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:24.944046    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwsrm_76693f47-85b7-4686-aac5-1c24d6ba4627/dns-node-resolver/0.log"
Jul 29 12:00:25.164564 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:25.164482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk" event={"ID":"27360541-e32e-4218-8151-ec26a51f72ef","Type":"ContainerStarted","Data":"e2bed82d48e698ca2b0a9be6b5f789b801eb58591ccf1c2846ca548a8d776cfe"}
Jul 29 12:00:25.164564 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:25.164552    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:25.164814 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:25.164578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk" event={"ID":"27360541-e32e-4218-8151-ec26a51f72ef","Type":"ContainerStarted","Data":"76bd3dd40edab7db4ce8a53bfdff40585423317ce6928a699767996bab5a037b"}
Jul 29 12:00:25.182207 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:25.182152    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk" podStartSLOduration=1.182108873 podStartE2EDuration="1.182108873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 12:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:00:25.1793088 +0000 UTC m=+5477.081537199" watchObservedRunningTime="2026-07-29 12:00:25.182108873 +0000 UTC m=+5477.084337273"
Jul 29 12:00:25.484477 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:25.484450    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8mdt_20587f8c-a3a2-48cf-b39d-a7a98a55c2d3/node-ca/0.log"
Jul 29 12:00:26.477522 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:26.477494    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s5cqz_5b0ba4af-fd88-4838-ae7a-2ce433e29b4a/serve-healthcheck-canary/0.log"
Jul 29 12:00:26.819754 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:26.819683    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccdw4_8e611c33-c2c7-49f5-965e-116fa6ae10e4/kube-rbac-proxy/0.log"
Jul 29 12:00:26.836037 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:26.836019    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccdw4_8e611c33-c2c7-49f5-965e-116fa6ae10e4/exporter/0.log"
Jul 29 12:00:26.853156 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:26.853139    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccdw4_8e611c33-c2c7-49f5-965e-116fa6ae10e4/extractor/0.log"
Jul 29 12:00:29.297043 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:29.297018    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-n9nqr_97115e08-1ec0-438d-8d84-15792117f185/server/0.log"
Jul 29 12:00:29.887601 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:29.887566    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vvhm9_2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c/s3-init/0.log"
Jul 29 12:00:29.900295 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:29.900271    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vvhm9_2e5234fa-dc2c-44ed-aa70-2a4ab10a9d4c/download-hf-model/0.log"
Jul 29 12:00:29.926381 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:29.926348    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rnwlj_7607eeee-1e7e-461f-9580-0a2f65efa14e/s3-tls-init-custom/0.log"
Jul 29 12:00:29.941357 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:29.941330    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rnwlj_7607eeee-1e7e-461f-9580-0a2f65efa14e/download-hf-model/0.log"
Jul 29 12:00:30.043733 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:30.043705    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9pt74_6159ef75-c8ac-4cc6-ac35-880ac8279972/s3-tls-init-serving/0.log"
Jul 29 12:00:30.058392 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:30.058373    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9pt74_6159ef75-c8ac-4cc6-ac35-880ac8279972/download-hf-model/0.log"
Jul 29 12:00:30.083405 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:30.083382    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-4fv2c_a94dfa90-f765-4ffa-90c4-330a38400d94/seaweedfs/0.log"
Jul 29 12:00:30.101798 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:30.101774    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-j5mmj_11ef9487-7446-4b44-b49d-b7d51e045db5/seaweedfs-tls-custom/0.log"
Jul 29 12:00:30.121276 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:30.121248    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-rwx5k_3e94fa99-8212-4ece-b8c8-5638fdc047af/seaweedfs-tls-serving/0.log"
Jul 29 12:00:31.180791 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:31.180760    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fjdcv/perf-node-gather-daemonset-pdsjk"
Jul 29 12:00:35.571005 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.570978    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/kube-multus-additional-cni-plugins/0.log"
Jul 29 12:00:35.588833 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.588796    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/egress-router-binary-copy/0.log"
Jul 29 12:00:35.605952 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.605929    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/cni-plugins/0.log"
Jul 29 12:00:35.626072 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.626049    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/bond-cni-plugin/0.log"
Jul 29 12:00:35.646044 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.646026    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/routeoverride-cni/0.log"
Jul 29 12:00:35.665725 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.665709    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/whereabouts-cni-bincopy/0.log"
Jul 29 12:00:35.686013 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.685994    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xq7m7_742e70a2-c439-4669-b404-fb48d7c4a419/whereabouts-cni/0.log"
Jul 29 12:00:35.752058 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.752015    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h45jj_9154bc9f-fb46-4c80-a45c-a886603cddee/kube-multus/0.log"
Jul 29 12:00:35.795506 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.795439    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9tsjs_de60eff5-1df5-4c1b-b19b-3ca6c0f2479a/network-metrics-daemon/0.log"
Jul 29 12:00:35.809343 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:35.809319    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9tsjs_de60eff5-1df5-4c1b-b19b-3ca6c0f2479a/kube-rbac-proxy/0.log"
Jul 29 12:00:36.530669 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.530642    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-controller/0.log"
Jul 29 12:00:36.546665 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.546641    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/1.log"
Jul 29 12:00:36.546798 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.546670    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovn-acl-logging/0.log"
Jul 29 12:00:36.561223 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.561199    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/kube-rbac-proxy-node/0.log"
Jul 29 12:00:36.580151 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.580109    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/kube-rbac-proxy-ovn-metrics/0.log"
Jul 29 12:00:36.599385 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.599362    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/northd/0.log"
Jul 29 12:00:36.617751 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.617723    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/nbdb/0.log"
Jul 29 12:00:36.640835 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.640817    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/sbdb/0.log"
Jul 29 12:00:36.704050 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:36.704028    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k49lk_1ef0b97e-f94b-497d-abb3-ab331d70b1cc/ovnkube-controller/0.log"
Jul 29 12:00:37.988822 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:37.988791    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-89sxt_df2f6da7-779a-4c19-a683-7d97b8d6eac2/network-check-target-container/0.log"
Jul 29 12:00:38.820847 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:38.820819    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-f7fj6_dd683971-80ad-4673-9fd9-1e1fdd69cda4/iptables-alerter/0.log"
Jul 29 12:00:39.453595 ip-10-0-134-91 kubenswrapper[2551]: I0729 12:00:39.453565    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vh8lq_375473e0-3d80-476e-8c4e-84205706af05/tuned/0.log"