Jul 28 15:30:54.009648 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 15:30:54.009661 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 15:30:54.009670 ip-10-0-140-97 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 15:30:54.010000 ip-10-0-140-97 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 15:31:04.258835 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 15:31:04.258850 ip-10-0-140-97 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dd6e67e915be4670902141810f5b6c2c --
Jul 28 15:33:05.500546 ip-10-0-140-97 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 15:33:05.926783 ip-10-0-140-97 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:05.926783 ip-10-0-140-97 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 15:33:05.926783 ip-10-0-140-97 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:05.926783 ip-10-0-140-97 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 15:33:05.926783 ip-10-0-140-97 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 15:33:05.927621 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.927519    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 15:33:05.931983 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931962    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.931983 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931978    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.931983 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931983    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.931983 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931988    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.931983 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931992    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.931996    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932001    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932005    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932010    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932014    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932017    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932021    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932025    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932029    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932033    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932037    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932040    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932044    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932047    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932051    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932055    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932059    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932063    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.932294 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932069    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932075    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932080    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932084    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932089    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932093    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932097    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932101    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932105    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932109    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932113    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932117    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932122    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932144    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932149    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932155    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932161    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932165    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932171    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932176    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.933058 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932180    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932187    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932192    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932197    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932202    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932207    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932211    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932215    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932219    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932223    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932227    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932232    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932236    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932240    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932244    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932248    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932252    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932256    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932261    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.933559 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932269    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932275    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932280    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932284    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932288    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932292    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932296    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932300    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932304    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932308    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932313    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932316    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932321    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932326    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932330    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932334    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932339    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932343    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932347    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932351    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.934040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932356    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932359    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932364    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932367    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932371    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932375    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932379    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932383    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932387    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932392    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932396    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932401    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932405    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932409    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932414    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932418    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.934602 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.932422    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.935761 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935743    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.935761 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935757    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.935761 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935762    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.935761 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935766    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935770    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935775    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935778    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935782    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935787    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935791    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935795    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935798    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935803    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935806    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935810    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935814    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935818    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935822    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935826    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935830    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935834    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935837    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935841    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.936001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935845    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935849    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935853    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935857    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935861    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935865    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935868    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935872    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935878    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935884    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935889    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935894    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935899    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935903    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935908    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935914    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935918    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935922    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935927    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.936822 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935932    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935936    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935940    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935944    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935948    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935952    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935958    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935962    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935967    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935970    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935975    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935979    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935983    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935987    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935991    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.935998    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936003    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936007    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936013    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936017    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.937556 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936022    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936027    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936031    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936036    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936040    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936045    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936049    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936054    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936058    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936065    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936069    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936074    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936079    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936083    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936087    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936091    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936095    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936099    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936103    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936108    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.938047 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936112    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936116    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936121    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936144    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936149    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936153    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936157    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936161    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936165    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936169    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936174    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936178    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936182    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936186    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936190    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936197    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.936202    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937508    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937525    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937535    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.938553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937673    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937682    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937685    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937689    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937691    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937694    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937697    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937700    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937703    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937706    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937708    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937711    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937713    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937716    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937719    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937721    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937724    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937726    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937729    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937732    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.939035 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937734    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937736    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937739    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937741    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937744    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937746    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937748    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937751    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937756    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937760    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937764    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937767    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937770    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937774    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937777    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937780    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937783    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937786    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.939534 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937790    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937793    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937796    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937799    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937802    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937804    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937807    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937809    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937812    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937814    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937817    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937820    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937822    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937825    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937827    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937830    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937832    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937835    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937838    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.939971 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937840    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937845    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937848    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937850    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937853    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937857    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937861    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937865    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937867    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937870    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937873    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937875    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937878    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937880    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937882    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937885    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937887    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937890    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937894    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937897    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.940453 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937899    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937902    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937904    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937907    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937909    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937912    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937914    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937917    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937919    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937921    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937924    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937926    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937929    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937931    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937933    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937936    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937938    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937944    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.937947    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938022    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938030    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 15:33:05.940973 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938038    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938042    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938046    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938050    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938054    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938058    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938061    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938064    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938068    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938071    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938074    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938076    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938079    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938082    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938085    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938089    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938094    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938097    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938100    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938103    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938106    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938110    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938113    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 15:33:05.941551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938116    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938119    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938121    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938140    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938143    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938148    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938151    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938155    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938158    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938161    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938163    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938166    2580 flags.go:64] FLAG: --enable-server="true"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938169    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938174    2580 flags.go:64] FLAG: --event-burst="100"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938177    2580 flags.go:64] FLAG: --event-qps="50"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938179    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938182    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938185    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938189    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938191    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938194    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938197    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938199    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938202    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938205    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 15:33:05.942096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938209    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938212    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938214    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938217    2580 flags.go:64] FLAG: --feature-gates=""
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938221    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938223    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938226    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938229    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938232    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938235    2580 flags.go:64] FLAG: --help="false"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938238    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-97.ec2.internal"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938241    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938245    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938248    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938251    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938254    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938257    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938260    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938263    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938266    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938268    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938271    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938274    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938276    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 15:33:05.942729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938279    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938282    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938284    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938287    2580 flags.go:64] FLAG: --lock-file=""
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938290    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938292    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938296    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938301    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938304    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938307    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938310    2580 flags.go:64] FLAG: --logging-format="text"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938312    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938315    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938318    2580 flags.go:64] FLAG: --manifest-url=""
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938321    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938327    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938329    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938333    2580 flags.go:64] FLAG: --max-pods="110"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938336    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938339    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938343    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938346    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938349    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938352    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938355    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 15:33:05.943329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938362    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938365    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938368    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938371    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938374    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938379    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938382    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938384    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938387    2580 flags.go:64] FLAG: --port="10250"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938390    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938392    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006da01f7b5a686e6"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938395    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938398    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938401    2580 flags.go:64] FLAG: --register-node="true"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938404    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938408    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938411    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938414    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938417    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938420    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938423    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938426    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938429    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938433    2580 flags.go:64] FLAG: --runonce="false"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938435    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 15:33:05.943925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938438    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938441    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938445    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938448    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938451    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938454    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938457    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938459    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938462    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938465    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938468    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938471    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938473    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938476    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938482    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938484    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938487    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938491    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938493    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938496    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938499    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938502    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938505    2580 flags.go:64] FLAG: --v="2"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938509    2580 flags.go:64] FLAG: --version="false"
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938512    2580 flags.go:64] FLAG: --vmodule=""
Jul 28 15:33:05.944598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938517    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.938520    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938658    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938662    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938666    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938670    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938673    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938676    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938679    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938683    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938685    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938688    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938691    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938693    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938696    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938699    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938702    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938706    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938710    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938713    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.945219 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938716    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938718    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938720    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938723    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938725    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938728    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938730    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938732    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938735    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938738    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938740    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938742    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938745    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938748    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938750    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938753    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938755    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938758    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938761    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938763    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.945696 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938766    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938769    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938771    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938774    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938776    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938779    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938781    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938784    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938786    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938789    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938792    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938794    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938798    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938802    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938804    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938807    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938810    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938812    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938815    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.946202 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938818    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938820    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938823    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938826    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938828    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938831    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938833    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938838    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938841    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938844    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938847    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938850    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938853    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938855    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938858    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938860    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938863    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938865    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938868    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.946661 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938870    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938873    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938875    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938878    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938881    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938883    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938886    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938888    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938890    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938893    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938895    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938898    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938900    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938902    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938905    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938907    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938910    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938912    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938915    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938917    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.947118 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938920    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938923    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.938926    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.939543    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.946463    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.946482    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.946500    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.946506    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946560    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946565    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946568    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946571    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946573    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946576    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946579    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.947607 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946581    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946585    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946587    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946590    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946592    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946595    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946597    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946601    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946605    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946609    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946612    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946615    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946618    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946620    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946623    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946625    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946628    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946630    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946633    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.947985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946636    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946639    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946643    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946647    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946650    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946652    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946655    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946658    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946660    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946663    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946665    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946668    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946671    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946674    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946676    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946679    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946682    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946684    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946687    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946689    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.948525 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946691    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946694    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946696    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946699    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946701    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946704    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946707    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946712    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946716    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946720    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946723    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946725    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946728    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946731    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946734    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946738    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946740    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946743    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.949001 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946746    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946748    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946751    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946753    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946756    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946759    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946761    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946764    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946766    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946768    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946771    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946773    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946776    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946778    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946781    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946783    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946786    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946788    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946791    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946793    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.949479 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946796    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946799    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946801    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946804    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946806    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946809    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946811    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946814    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946817    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946820    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946823    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946826    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946828    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.946836    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:05.949966 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946958    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946963    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946966    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946969    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946972    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946975    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946977    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946980    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946982    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946985    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946987    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946990    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946992    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946995    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.946997    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947000    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947003    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947006    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947008    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947011    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 15:33:05.950356 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947013    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947016    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947020    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947023    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947026    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947029    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947032    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947034    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947038    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947041    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947043    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947046    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947050    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947052    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947055    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947060    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947062    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947065    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 15:33:05.950839 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947067    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947070    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947073    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947075    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947078    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947080    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947083    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947085    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947088    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947090    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947093    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947095    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947098    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947100    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947103    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947105    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947108    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947110    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947113    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947115    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 15:33:05.951309 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947118    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947120    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947142    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947146    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947150    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947152    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947155    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947157    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947160    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947163    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947165    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947168    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947170    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947173    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947175    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947177    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947180    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947182    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947185    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947187    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 15:33:05.951778 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947190    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947193    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947195    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947197    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947200    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947202    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947205    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947207    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947210    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947212    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947215    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947217    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947219    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947222    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947224    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947227    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947230    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947234    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947237    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 15:33:05.952331 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:05.947239    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.947244    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.947371    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.949369    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.950633    2580 server.go:1019] "Starting client certificate rotation"
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.950735    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:33:05.952798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.950766    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 15:33:05.953119 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.953104    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:33:05.954324 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.954309    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 15:33:05.962942 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.962925    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 28 15:33:05.964849 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.964830    2580 log.go:25] "Validated CRI v1 image API"
Jul 28 15:33:05.965844 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.965828    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 15:33:05.968138 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.968105    2580 fs.go:135] Filesystem UUIDs: map[1b6724eb-ffd6-4828-ade2-1c594b8d03c8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e0db1e14-7b26-44ee-aac2-a54f7d32bc14:/dev/nvme0n1p4]
Jul 28 15:33:05.968184 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.968140    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 15:33:05.975274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.975110    2580 manager.go:216] Machine: {Timestamp:2026-07-28 15:33:05.973825552 +0000 UTC m=+0.342478459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110539 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2d4f9378ea2e4cd72f65d03f9560 SystemUUID:ec2f2d4f-9378-ea2e-4cd7-2f65d03f9560 BootID:dd6e67e9-15be-4670-9021-41810f5b6c2c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:59:38:cd:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:59:38:cd:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:89:21:ea:e3:6c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 15:33:05.975274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.975257    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 15:33:05.975503 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.975399    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 15:33:05.978877 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.978828    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 15:33:05.979058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.978879    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 15:33:05.979685 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.979672    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 15:33:05.979727 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.979687    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 15:33:05.979727 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.979698    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:33:05.980364 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.980352    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 15:33:05.981174 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.981162    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 15:33:05.983021 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.983007    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:33:05.983483 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.983470    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 15:33:05.984099 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984089    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 15:33:05.984156 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984113    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 15:33:05.984156 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984153    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 15:33:05.984232 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984165    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 28 15:33:05.984232 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984178    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 15:33:05.984733 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984719    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:33:05.984799 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.984741    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 15:33:05.987190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.987163    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 15:33:05.989109 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.989094    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 15:33:05.989200 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.989141    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 15:33:05.990617 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990602    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 15:33:05.990668 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990624    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 15:33:05.990668 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990634    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 15:33:05.990668 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990643    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 15:33:05.990668 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990658    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 15:33:05.990668 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990666    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990675    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990681    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990690    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990696    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990718    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 15:33:05.990807 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.990725    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 15:33:05.992305 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.992290    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 15:33:05.992377 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.992307    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 15:33:05.996017 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.996002    2580 server.go:1268] "Started kubelet"
Jul 28 15:33:05.996108 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.996081    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 15:33:05.996179 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.996086    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 15:33:05.996229 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.996179    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 15:33:05.997154 ip-10-0-140-97 systemd[1]: Started Kubernetes Kubelet.
Jul 28 15:33:05.997864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.997819    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 15:33:05.999342 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.999059    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 15:33:05.999342 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:05.999243    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 15:33:05.999527 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:05.999476    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:33:05.999586 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:05.999471    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 15:33:06.000157 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.000142    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 15:33:06.001435 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.001408    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 15:33:06.005513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.005493    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 15:33:06.005595 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.005558    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 15:33:06.005656 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.005647    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.005877 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.004825    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c67df33f90ff96  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-28 15:33:05.995972502 +0000 UTC m=+0.364625470,LastTimestamp:2026-07-28 15:33:05.995972502 +0000 UTC m=+0.364625470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}"
Jul 28 15:33:06.006569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006095    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 15:33:06.006569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006324    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 15:33:06.006569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006340    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 15:33:06.006569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006558    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 15:33:06.006569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006572    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 15:33:06.006879 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006743    2580 factory.go:55] Registering systemd factory
Jul 28 15:33:06.006879 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.006761    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 28 15:33:06.008924 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.008790    2580 factory.go:153] Registering CRI-O factory
Jul 28 15:33:06.008924 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.008826    2580 factory.go:223] Registration of the crio container factory successfully
Jul 28 15:33:06.009043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.008923    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 15:33:06.009043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.008951    2580 factory.go:103] Registering Raw factory
Jul 28 15:33:06.009043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.008968    2580 manager.go:1196] Started watching for new ooms in manager
Jul 28 15:33:06.010025 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.010000    2580 manager.go:349] Starting recovery of all containers
Jul 28 15:33:06.016940 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.016751    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 15:33:06.016940 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.016862    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 15:33:06.017773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.017707    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 15:33:06.023108 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.023087    2580 manager.go:354] Recovery completed
Jul 28 15:33:06.027445 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.027431    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.029550 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.029528    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.029602 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.029565    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.029602 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.029575    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.030186 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030172    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 15:33:06.030186 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030183    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.030325 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030184    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 15:33:06.030325 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030252    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 15:33:06.030872 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030856    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.030984 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030883    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.030984 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.030896    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.032048 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.032035    2580 policy_none.go:49] "None policy: Start"
Jul 28 15:33:06.032176 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.032054    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 15:33:06.032176 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.032069    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 15:33:06.033141 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.033112    2580 policy_none.go:47] "Start"
Jul 28 15:33:06.040978 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.040889    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c67df341915dde  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-28 15:33:06.02955107 +0000 UTC m=+0.398203979,LastTimestamp:2026-07-28 15:33:06.02955107 +0000 UTC m=+0.398203979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}"
Jul 28 15:33:06.058019 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.057937    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c67df34191a6af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-97.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-28 15:33:06.029569711 +0000 UTC m=+0.398222617,LastTimestamp:2026-07-28 15:33:06.029569711 +0000 UTC m=+0.398222617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}"
Jul 28 15:33:06.058409 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.058390    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5zr55"
Jul 28 15:33:06.070768 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.070747    2580 manager.go:337] "Starting Device Plugin manager"
Jul 28 15:33:06.070892 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.070787    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 15:33:06.070892 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.070797    2580 server.go:87] "Starting device plugin registration server"
Jul 28 15:33:06.071079 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.071069    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 15:33:06.071151 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.071083    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 15:33:06.071252 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.071233    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 15:33:06.071356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.071338    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 15:33:06.071356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.071349    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 15:33:06.071912 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.071894    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 15:33:06.071978 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.071940    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.073368 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.073351    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5zr55"
Jul 28 15:33:06.081392 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.081318    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c67df34191cbef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-97.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-28 15:33:06.029579247 +0000 UTC m=+0.398232154,LastTimestamp:2026-07-28 15:33:06.029579247 +0000 UTC m=+0.398232154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}"
Jul 28 15:33:06.171911 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.171875    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.173227 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.173179    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.173227 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.173215    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.173325 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.173236    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.173325 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.173269    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.180295 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.180251    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 15:33:06.180295 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.180293    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 15:33:06.180400 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.180313    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 15:33:06.180400 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.180350    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 15:33:06.184813 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.184789    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:06.185654 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.185639    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.185699 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.185662    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-97.ec2.internal\": node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.281205 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.281166    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"]
Jul 28 15:33:06.281380 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.281244    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.282773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.282757    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.282914 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.282795    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.282914 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.282805    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.284018 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284005    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.284169 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.284274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284194    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.284773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284761    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.284828 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284783    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.284828 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.284794    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.285854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.285815    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.285941 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.285874    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.285941 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.285892    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.288378 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.288361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.288463 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.288389    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:06.289121 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.289104    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:06.289121 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.289157    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:06.289121 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.289172    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:06.307745 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.307718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.307847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.307747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.307847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.307773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/969cac25ca034fa30dfc485a33f781d6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"969cac25ca034fa30dfc485a33f781d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.311483 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.311466    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.313881 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.313867    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-97.ec2.internal\" not found" node="ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.318690 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.318675    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-97.ec2.internal\" not found" node="ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408273 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/969cac25ca034fa30dfc485a33f781d6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"969cac25ca034fa30dfc485a33f781d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408273 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408466 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408466 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/969cac25ca034fa30dfc485a33f781d6-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"969cac25ca034fa30dfc485a33f781d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408466 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.408466 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.408368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee9910ae154d972e1b5202c2b998fc72-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"ee9910ae154d972e1b5202c2b998fc72\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.412353 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.412330    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.512474 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.512395    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.612915 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.612890    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.616028 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.616006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.622224 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.622198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:06.713277 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.713253    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.813648 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.813571    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.913914 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:06.913880    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:06.928729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.928708    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:06.950212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.950188    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 15:33:06.950365 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.950326    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:33:06.950398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:06.950363    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 15:33:07.014416 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:07.014386    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:07.077529 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.077417    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:28:06 +0000 UTC" deadline="2028-04-16 15:36:05.170694094 +0000 UTC"
Jul 28 15:33:07.077529 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.077474    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15072h2m58.093223994s"
Jul 28 15:33:07.101584 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.101558    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 15:33:07.103069 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.103040    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 15:33:07.103069 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.103080    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 15:33:07.103258 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.103091    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID"
Jul 28 15:33:07.115023 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:07.115000    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found"
Jul 28 15:33:07.121987 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.121970    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:07.197557 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.197533    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:07.206859 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.206837    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:07.212553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:07.212527    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9910ae154d972e1b5202c2b998fc72.slice/crio-d7f7cc4b0893da12ca3ade5ec5c991328785e2c863a1a6a92eafa94e70ad3569 WatchSource:0}: Error finding container d7f7cc4b0893da12ca3ade5ec5c991328785e2c863a1a6a92eafa94e70ad3569: Status 404 returned error can't find the container with id d7f7cc4b0893da12ca3ade5ec5c991328785e2c863a1a6a92eafa94e70ad3569
Jul 28 15:33:07.212781 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:07.212763    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969cac25ca034fa30dfc485a33f781d6.slice/crio-d8fb3a1aec5c311ae05da8c48b5a3deb3e647a25e6e3fe69fdafae255c337550 WatchSource:0}: Error finding container d8fb3a1aec5c311ae05da8c48b5a3deb3e647a25e6e3fe69fdafae255c337550: Status 404 returned error can't find the container with id d8fb3a1aec5c311ae05da8c48b5a3deb3e647a25e6e3fe69fdafae255c337550
Jul 28 15:33:07.217750 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.217733    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:33:07.226812 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.226791    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:33:07.227752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.227739    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal"
Jul 28 15:33:07.271229 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.271192    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 15:33:07.947480 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.947452    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:07.985324 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.985293    2580 apiserver.go:52] "Watching apiserver"
Jul 28 15:33:07.999840 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:07.999812    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 15:33:08.000327 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.000299    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-sxwcf","openshift-network-operator/iptables-alerter-rcwf8","openshift-ovn-kubernetes/ovnkube-node-2dc94","kube-system/global-pull-secret-syncer-jstzl","kube-system/konnectivity-agent-dbx6n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7","openshift-cluster-node-tuning-operator/tuned-m6mgg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal","openshift-multus/multus-drbzm","openshift-multus/network-metrics-daemon-krkl4","openshift-network-diagnostics/network-check-target-bnw82","kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal","openshift-dns/node-resolver-z7bcq","openshift-image-registry/node-ca-qn2n4"]
Jul 28 15:33:08.002187 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.002162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.003671 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.003647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.006321 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.006088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.006321 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.006170    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 15:33:08.007348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.007324    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fxmrb\""
Jul 28 15:33:08.007628 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.007604    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.007700 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.007668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.007765 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.007737    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:08.008116 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.008093    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.008226 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.008161    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.008621 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.008160    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.008726 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.008681    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnszd\""
Jul 28 15:33:08.009504 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.009483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.011408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.011387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.013802 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.013178    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.014834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.014795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 15:33:08.015006 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.014986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.015088 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015058    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 15:33:08.015338 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.015394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-systemd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015448 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-var-lib-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015533 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovn-node-metrics-cert\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015533 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-modprobe-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015533 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-run\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015629 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-sys\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015629 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9x7\" (UniqueName: \"kubernetes.io/projected/adee8374-4c30-41b2-a493-6274dfa19bc5-kube-api-access-fp9x7\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015629 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015713 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-netd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015713 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-kubelet-config\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.015798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-conf\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-systemd\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015876 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-tuned\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.015876 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119e856f-abfc-42cd-9d2c-052ed52df947-hosts-file\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.015936 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.015979 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.016015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.015971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.016050 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysconfig\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.016142 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-ovn\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.016195 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-script-lib\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.016236 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkkz\" (UniqueName: \"kubernetes.io/projected/1eb13ef3-41ab-45ce-9a59-fc57643325f7-kube-api-access-6fkkz\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.016323 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wx9\" (UniqueName: \"kubernetes.io/projected/119e856f-abfc-42cd-9d2c-052ed52df947-kube-api-access-z5wx9\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.016370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.016307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-slash\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.017475 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-kubernetes\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.017587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-host\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.017587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-node-log\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.017587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.017587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-lib-modules\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.017587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-var-lib-kubelet\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-kubelet\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-etc-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-bin\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 15:33:08.017837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-env-overrides\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-dbus\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sjfl2\""
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119e856f-abfc-42cd-9d2c-052ed52df947-tmp-dir\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-netns\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.017977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-log-socket\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.018002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-config\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.018022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-tmp\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.018120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.018056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-systemd-units\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.019215 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.019199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.021318 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.021300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.021414 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.021372    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:08.022916 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.022897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:08.022991 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.022962    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:08.023911 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.023894    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.023911 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.023901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.024053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.023935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 15:33:08.024517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.024179    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 15:33:08.024517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.024430    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.024517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.024490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 15:33:08.024645 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.024610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 15:33:08.024723 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.024705    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025576    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sjkrw\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025670    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025674    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.025937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.025908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fpxzh\""
Jul 28 15:33:08.026429 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.026341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w4mlq\""
Jul 28 15:33:08.026574 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.026553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pfs8z\""
Jul 28 15:33:08.026574 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.026568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-628gw\""
Jul 28 15:33:08.027485 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.027371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 15:33:08.029472 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.029450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cq7xp\""
Jul 28 15:33:08.031789 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.031766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.032085 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.032069    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 15:33:08.032455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.032439    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 15:33:08.046752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.046728    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 15:33:08.073594 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.073565    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tbjtf"
Jul 28 15:33:08.083303 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.083276    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tbjtf"
Jul 28 15:33:08.107304 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.107275    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 15:33:08.118962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.118927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.118977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-sys-fs\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-conf-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119155 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-ovn\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-script-lib\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkkz\" (UniqueName: \"kubernetes.io/projected/1eb13ef3-41ab-45ce-9a59-fc57643325f7-kube-api-access-6fkkz\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wx9\" (UniqueName: \"kubernetes.io/projected/119e856f-abfc-42cd-9d2c-052ed52df947-kube-api-access-z5wx9\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cni-binary-copy\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-socket-dir-parent\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119586 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.119586 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26tb\" (UniqueName: \"kubernetes.io/projected/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-kube-api-access-v26tb\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.119586 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-slash\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.119586 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-host\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.119763 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-os-release\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119763 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-multus-certs\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119763 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-lib-modules\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.119848 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-var-lib-kubelet\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.119848 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-iptables-alerter-script\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.119906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-bin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlsd7\" (UniqueName: \"kubernetes.io/projected/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-kube-api-access-vlsd7\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.119976 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-dbus\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.119976 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119e856f-abfc-42cd-9d2c-052ed52df947-tmp-dir\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.119976 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzt2\" (UniqueName: \"kubernetes.io/projected/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-kube-api-access-zjzt2\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.120067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.119986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cnibin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.120067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-netns\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-log-socket\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120057    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-device-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.120245 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:08.120245 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5fs\" (UniqueName: \"kubernetes.io/projected/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-kube-api-access-tz5fs\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.120245 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-script-lib\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120245 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-log-socket\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-sys\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-host\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-run-netns\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-var-lib-kubelet\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-dbus\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-ovn\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-slash\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtnz\" (UniqueName: \"kubernetes.io/projected/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-kube-api-access-nrtnz\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-sys\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-lib-modules\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120423 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-netd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119e856f-abfc-42cd-9d2c-052ed52df947-tmp-dir\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-netd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-kubelet-config\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-conf\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-systemd\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-tuned\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-registration-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-serviceca\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysconfig\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-host-slash\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-system-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.120945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysconfig\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.120982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-conf\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121031    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-hostroot\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3734d978-9bc8-4025-99ee-f847754332e5-kubelet-config\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-systemd\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-etc-kubernetes\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-system-cni-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-kubernetes\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff52c32e-08f8-4417-bda9-9eddd9a13efc-agent-certs\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-kubernetes\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-k8s-cni-cncf-io\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-multus\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-node-log\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.121453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-kubelet\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-node-log\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-etc-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-kubelet\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.121616    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-bin\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-etc-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-env-overrides\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-host-cni-bin\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.121747    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:08.62171064 +0000 UTC m=+2.990363553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff52c32e-08f8-4417-bda9-9eddd9a13efc-konnectivity-ca\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-host\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-netns\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121918    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-sysctl-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.122152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-config\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-tmp\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.121987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmzk\" (UniqueName: \"kubernetes.io/projected/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kube-api-access-fsmzk\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-os-release\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119e856f-abfc-42cd-9d2c-052ed52df947-hosts-file\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-env-overrides\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-socket-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-systemd-units\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-systemd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-var-lib-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/119e856f-abfc-42cd-9d2c-052ed52df947-hosts-file\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-systemd\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-systemd-units\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovn-node-metrics-cert\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-var-lib-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-modprobe-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-run\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.122863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-modprobe-d\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9x7\" (UniqueName: \"kubernetes.io/projected/adee8374-4c30-41b2-a493-6274dfa19bc5-kube-api-access-fp9x7\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1eb13ef3-41ab-45ce-9a59-fc57643325f7-run-openvswitch\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-kubelet\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-daemon-config\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/adee8374-4c30-41b2-a493-6274dfa19bc5-run\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovnkube-config\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.123651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.122501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cnibin\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.125055 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.125029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-tmp\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.125180 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.125079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/adee8374-4c30-41b2-a493-6274dfa19bc5-etc-tuned\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.128375 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.128350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1eb13ef3-41ab-45ce-9a59-fc57643325f7-ovn-node-metrics-cert\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.153088 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.153049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wx9\" (UniqueName: \"kubernetes.io/projected/119e856f-abfc-42cd-9d2c-052ed52df947-kube-api-access-z5wx9\") pod \"node-resolver-z7bcq\" (UID: \"119e856f-abfc-42cd-9d2c-052ed52df947\") " pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.154606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.154557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkkz\" (UniqueName: \"kubernetes.io/projected/1eb13ef3-41ab-45ce-9a59-fc57643325f7-kube-api-access-6fkkz\") pod \"ovnkube-node-2dc94\" (UID: \"1eb13ef3-41ab-45ce-9a59-fc57643325f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.157349 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.157323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9x7\" (UniqueName: \"kubernetes.io/projected/adee8374-4c30-41b2-a493-6274dfa19bc5-kube-api-access-fp9x7\") pod \"tuned-m6mgg\" (UID: \"adee8374-4c30-41b2-a493-6274dfa19bc5\") " pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.185699 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.185641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"ee9910ae154d972e1b5202c2b998fc72","Type":"ContainerStarted","Data":"d7f7cc4b0893da12ca3ade5ec5c991328785e2c863a1a6a92eafa94e70ad3569"}
Jul 28 15:33:08.187168 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.187118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" event={"ID":"969cac25ca034fa30dfc485a33f781d6","Type":"ContainerStarted","Data":"d8fb3a1aec5c311ae05da8c48b5a3deb3e647a25e6e3fe69fdafae255c337550"}
Jul 28 15:33:08.223098 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223098 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtnz\" (UniqueName: \"kubernetes.io/projected/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-kube-api-access-nrtnz\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.223098 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-registration-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-serviceca\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-host-slash\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-system-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-registration-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-hostroot\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-etc-kubernetes\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-system-cni-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-system-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff52c32e-08f8-4417-bda9-9eddd9a13efc-agent-certs\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-k8s-cni-cncf-io\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-multus\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-hostroot\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-cni-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff52c32e-08f8-4417-bda9-9eddd9a13efc-konnectivity-ca\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-host\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-netns\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmzk\" (UniqueName: \"kubernetes.io/projected/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kube-api-access-fsmzk\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-os-release\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-socket-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-kubelet\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-daemon-config\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cnibin\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-sys-fs\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.223971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-conf-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-serviceca\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cni-binary-copy\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-socket-dir-parent\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26tb\" (UniqueName: \"kubernetes.io/projected/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-kube-api-access-v26tb\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-host\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.223957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-os-release\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-multus-certs\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224014    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-os-release\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-iptables-alerter-script\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-bin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlsd7\" (UniqueName: \"kubernetes.io/projected/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-kube-api-access-vlsd7\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-os-release\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzt2\" (UniqueName: \"kubernetes.io/projected/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-kube-api-access-zjzt2\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cnibin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-device-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.224752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-socket-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5fs\" (UniqueName: \"kubernetes.io/projected/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-kube-api-access-tz5fs\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-multus-certs\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff52c32e-08f8-4417-bda9-9eddd9a13efc-konnectivity-ca\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-socket-dir-parent\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-netns\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-host-slash\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-bin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-cni-multus\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-run-k8s-cni-cncf-io\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-device-dir\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.224976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-iptables-alerter-script\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cnibin\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.225536 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.225244    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-host-var-lib-kubelet\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-etc-kubernetes\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-system-cni-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225414    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-conf-dir\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163fe8a6-6224-4bb1-9a53-e2f67f35e234-sys-fs\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225528    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-cnibin\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.225583    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:08.725568787 +0000 UTC m=+3.094221682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.225603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.226089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.226075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff52c32e-08f8-4417-bda9-9eddd9a13efc-agent-certs\") pod \"konnectivity-agent-dbx6n\" (UID: \"ff52c32e-08f8-4417-bda9-9eddd9a13efc\") " pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.226753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.226735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-multus-daemon-config\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.226849 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.226829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-cni-binary-copy\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.237269 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.237249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmzk\" (UniqueName: \"kubernetes.io/projected/163fe8a6-6224-4bb1-9a53-e2f67f35e234-kube-api-access-fsmzk\") pod \"aws-ebs-csi-driver-node-fcjq7\" (UID: \"163fe8a6-6224-4bb1-9a53-e2f67f35e234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.237834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.237815    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtnz\" (UniqueName: \"kubernetes.io/projected/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-kube-api-access-nrtnz\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.243331 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.243305    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:08.243331 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.243333    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:08.243499 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.243346    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:08.243499 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.243468    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:08.743445671 +0000 UTC m=+3.112098589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:08.245909 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.245883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26tb\" (UniqueName: \"kubernetes.io/projected/3702206b-0647-4ea7-8cef-4fa6b7fa1fdd-kube-api-access-v26tb\") pod \"multus-additional-cni-plugins-sxwcf\" (UID: \"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd\") " pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.247937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.247913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlsd7\" (UniqueName: \"kubernetes.io/projected/0dfe30ab-2aa2-416f-a1d8-f72f0767044a-kube-api-access-vlsd7\") pod \"multus-drbzm\" (UID: \"0dfe30ab-2aa2-416f-a1d8-f72f0767044a\") " pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.249702 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.249680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzt2\" (UniqueName: \"kubernetes.io/projected/61cf79f4-d5f4-49fb-991a-1618cb42a3f6-kube-api-access-zjzt2\") pod \"iptables-alerter-rcwf8\" (UID: \"61cf79f4-d5f4-49fb-991a-1618cb42a3f6\") " pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.252356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.252334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5fs\" (UniqueName: \"kubernetes.io/projected/aeaab55a-2ceb-4543-a282-3a3ebebcb0af-kube-api-access-tz5fs\") pod \"node-ca-qn2n4\" (UID: \"aeaab55a-2ceb-4543-a282-3a3ebebcb0af\") " pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.318081 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.318041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m6mgg"
Jul 28 15:33:08.327315 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.327293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z7bcq"
Jul 28 15:33:08.336445 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.336244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:08.343452 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.343427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:08.350206 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.350161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drbzm"
Jul 28 15:33:08.357917 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.357887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxwcf"
Jul 28 15:33:08.364628 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.364605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7"
Jul 28 15:33:08.372588 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.372562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rcwf8"
Jul 28 15:33:08.378730 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.378704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qn2n4"
Jul 28 15:33:08.557643 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.557564    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 15:33:08.628211 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.628171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:08.628378 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.628310    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:08.628423 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.628383    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:09.628362094 +0000 UTC m=+3.997015006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:08.729316 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.729086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:08.729316 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.729264    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:08.729540 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.729341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:09.729321606 +0000 UTC m=+4.097974514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:08.829767 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:08.829687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:08.829925 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.829857    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:08.829925 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.829881    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:08.829925 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.829891    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:08.830040 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:08.829949    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:09.829931009 +0000 UTC m=+4.198583904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:09.016297 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.016271    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadee8374_4c30_41b2_a493_6274dfa19bc5.slice/crio-69faac987c3321b575d3f14d29c19d32beb88b3f28722c61369de4b3b3324b03 WatchSource:0}: Error finding container 69faac987c3321b575d3f14d29c19d32beb88b3f28722c61369de4b3b3324b03: Status 404 returned error can't find the container with id 69faac987c3321b575d3f14d29c19d32beb88b3f28722c61369de4b3b3324b03
Jul 28 15:33:09.018168 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.018015    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cf79f4_d5f4_49fb_991a_1618cb42a3f6.slice/crio-1f349ebba67652f15e0d888e91b79691c4f8e2182419c56cbb47e691fc6c68f6 WatchSource:0}: Error finding container 1f349ebba67652f15e0d888e91b79691c4f8e2182419c56cbb47e691fc6c68f6: Status 404 returned error can't find the container with id 1f349ebba67652f15e0d888e91b79691c4f8e2182419c56cbb47e691fc6c68f6
Jul 28 15:33:09.019108 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.018950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaab55a_2ceb_4543_a282_3a3ebebcb0af.slice/crio-cf9e5e131c4fcb63520de28f23c1a94895282a1d4d596b642a2267c5b24625e9 WatchSource:0}: Error finding container cf9e5e131c4fcb63520de28f23c1a94895282a1d4d596b642a2267c5b24625e9: Status 404 returned error can't find the container with id cf9e5e131c4fcb63520de28f23c1a94895282a1d4d596b642a2267c5b24625e9
Jul 28 15:33:09.022432 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.022391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119e856f_abfc_42cd_9d2c_052ed52df947.slice/crio-87fe4e8cddeeb5d7fd4304b9bf5dc982b70a49253b55e7b2107f8b7a23d688fd WatchSource:0}: Error finding container 87fe4e8cddeeb5d7fd4304b9bf5dc982b70a49253b55e7b2107f8b7a23d688fd: Status 404 returned error can't find the container with id 87fe4e8cddeeb5d7fd4304b9bf5dc982b70a49253b55e7b2107f8b7a23d688fd
Jul 28 15:33:09.024225 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.024186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff52c32e_08f8_4417_bda9_9eddd9a13efc.slice/crio-2b7fd05c55c2ec65fcb432f117ddeab2d441b240f6851f440112ec2a89c45ac9 WatchSource:0}: Error finding container 2b7fd05c55c2ec65fcb432f117ddeab2d441b240f6851f440112ec2a89c45ac9: Status 404 returned error can't find the container with id 2b7fd05c55c2ec65fcb432f117ddeab2d441b240f6851f440112ec2a89c45ac9
Jul 28 15:33:09.024832 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.024809    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb13ef3_41ab_45ce_9a59_fc57643325f7.slice/crio-5dc0d1d3e1bcb535fe482e27836815363537fb6b3a5d6a037ad7d0557cc681e3 WatchSource:0}: Error finding container 5dc0d1d3e1bcb535fe482e27836815363537fb6b3a5d6a037ad7d0557cc681e3: Status 404 returned error can't find the container with id 5dc0d1d3e1bcb535fe482e27836815363537fb6b3a5d6a037ad7d0557cc681e3
Jul 28 15:33:09.026157 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.025809    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3702206b_0647_4ea7_8cef_4fa6b7fa1fdd.slice/crio-c54ddf994ac65ca10067ef9eca3a9c263f029f5f23e0fb6d3cf943f94181648e WatchSource:0}: Error finding container c54ddf994ac65ca10067ef9eca3a9c263f029f5f23e0fb6d3cf943f94181648e: Status 404 returned error can't find the container with id c54ddf994ac65ca10067ef9eca3a9c263f029f5f23e0fb6d3cf943f94181648e
Jul 28 15:33:09.027886 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.026863    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfe30ab_2aa2_416f_a1d8_f72f0767044a.slice/crio-292b7efad21f5239244b159a2796f28106ade42e6f8927279be76528a605c046 WatchSource:0}: Error finding container 292b7efad21f5239244b159a2796f28106ade42e6f8927279be76528a605c046: Status 404 returned error can't find the container with id 292b7efad21f5239244b159a2796f28106ade42e6f8927279be76528a605c046
Jul 28 15:33:09.028040 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:33:09.027914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163fe8a6_6224_4bb1_9a53_e2f67f35e234.slice/crio-e1ca92d7f4582dd7a3dc16bac095d4753b4e3ec3c4ee618d94a2a53d6cc528cc WatchSource:0}: Error finding container e1ca92d7f4582dd7a3dc16bac095d4753b4e3ec3c4ee618d94a2a53d6cc528cc: Status 404 returned error can't find the container with id e1ca92d7f4582dd7a3dc16bac095d4753b4e3ec3c4ee618d94a2a53d6cc528cc
Jul 28 15:33:09.085758 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.085539    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:08 +0000 UTC" deadline="2028-02-08 22:02:41.636716028 +0000 UTC"
Jul 28 15:33:09.085758 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.085704    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13446h29m32.551014998s"
Jul 28 15:33:09.180739 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.180707    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:09.181010 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.180821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:09.181010 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.180874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:09.181010 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.180992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:09.190825 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.190797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" event={"ID":"969cac25ca034fa30dfc485a33f781d6","Type":"ContainerStarted","Data":"60a9ad533cf4f452c952d1e2d215f21bddb809d15cd3007170a775521d5477d6"}
Jul 28 15:33:09.191896 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.191875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drbzm" event={"ID":"0dfe30ab-2aa2-416f-a1d8-f72f0767044a","Type":"ContainerStarted","Data":"292b7efad21f5239244b159a2796f28106ade42e6f8927279be76528a605c046"}
Jul 28 15:33:09.192794 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.192774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn2n4" event={"ID":"aeaab55a-2ceb-4543-a282-3a3ebebcb0af","Type":"ContainerStarted","Data":"cf9e5e131c4fcb63520de28f23c1a94895282a1d4d596b642a2267c5b24625e9"}
Jul 28 15:33:09.193769 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.193752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rcwf8" event={"ID":"61cf79f4-d5f4-49fb-991a-1618cb42a3f6","Type":"ContainerStarted","Data":"1f349ebba67652f15e0d888e91b79691c4f8e2182419c56cbb47e691fc6c68f6"}
Jul 28 15:33:09.194727 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.194708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7" event={"ID":"163fe8a6-6224-4bb1-9a53-e2f67f35e234","Type":"ContainerStarted","Data":"e1ca92d7f4582dd7a3dc16bac095d4753b4e3ec3c4ee618d94a2a53d6cc528cc"}
Jul 28 15:33:09.195755 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.195737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"5dc0d1d3e1bcb535fe482e27836815363537fb6b3a5d6a037ad7d0557cc681e3"}
Jul 28 15:33:09.196782 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.196764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerStarted","Data":"c54ddf994ac65ca10067ef9eca3a9c263f029f5f23e0fb6d3cf943f94181648e"}
Jul 28 15:33:09.197784 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.197762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbx6n" event={"ID":"ff52c32e-08f8-4417-bda9-9eddd9a13efc","Type":"ContainerStarted","Data":"2b7fd05c55c2ec65fcb432f117ddeab2d441b240f6851f440112ec2a89c45ac9"}
Jul 28 15:33:09.200954 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.200930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7bcq" event={"ID":"119e856f-abfc-42cd-9d2c-052ed52df947","Type":"ContainerStarted","Data":"87fe4e8cddeeb5d7fd4304b9bf5dc982b70a49253b55e7b2107f8b7a23d688fd"}
Jul 28 15:33:09.202470 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.202449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m6mgg" event={"ID":"adee8374-4c30-41b2-a493-6274dfa19bc5","Type":"ContainerStarted","Data":"69faac987c3321b575d3f14d29c19d32beb88b3f28722c61369de4b3b3324b03"}
Jul 28 15:33:09.212532 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.212490    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" podStartSLOduration=2.212475012 podStartE2EDuration="2.212475012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:09.210460606 +0000 UTC m=+3.579113522" watchObservedRunningTime="2026-07-28 15:33:09.212475012 +0000 UTC m=+3.581127928"
Jul 28 15:33:09.635691 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.635638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:09.635876 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.635824    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:09.635934 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.635899    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.635881022 +0000 UTC m=+6.004533918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:09.736679 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.736640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:09.736858 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.736785    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:09.736858 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.736846    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.736827844 +0000 UTC m=+6.105480742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:09.837869 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:09.837252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:09.837869 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.837428    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:09.837869 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.837449    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:09.837869 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.837462    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:09.837869 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:09.837523    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:11.837505192 +0000 UTC m=+6.206158100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:10.086918 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:10.086803    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:28:08 +0000 UTC" deadline="2028-04-07 20:53:07.838535756 +0000 UTC"
Jul 28 15:33:10.086918 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:10.086865    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h19m57.751674783s"
Jul 28 15:33:10.181747 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:10.181714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:10.181911 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:10.181867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:10.207934 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:10.207897    2580 generic.go:356] "Generic (PLEG): container finished" podID="ee9910ae154d972e1b5202c2b998fc72" containerID="135e44056400fd44347ac4d2a92209c96c7cec95cfe0274cfad402fc1980f514" exitCode=0
Jul 28 15:33:10.209043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:10.209015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"ee9910ae154d972e1b5202c2b998fc72","Type":"ContainerDied","Data":"135e44056400fd44347ac4d2a92209c96c7cec95cfe0274cfad402fc1980f514"}
Jul 28 15:33:11.181865 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.181175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:11.181865 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.181313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:11.181865 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.181730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:11.181865 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.181824    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:11.230322 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.229661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"ee9910ae154d972e1b5202c2b998fc72","Type":"ContainerStarted","Data":"73fcddda5ab05a73071c4656757a1899ae99190ad46e78f3b61f2fd2868a9bc1"}
Jul 28 15:33:11.259390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.259021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" podStartSLOduration=4.258998975 podStartE2EDuration="4.258998975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:11.258297156 +0000 UTC m=+5.626950073" watchObservedRunningTime="2026-07-28 15:33:11.258998975 +0000 UTC m=+5.627651892"
Jul 28 15:33:11.659100 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.659013    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:11.659278 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.659181    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:11.659278 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.659251    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:15.659233807 +0000 UTC m=+10.027886703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:11.760381 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.759511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:11.760381 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.759700    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:11.760381 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.759758    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:15.759740427 +0000 UTC m=+10.128393349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:11.861071 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:11.860295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:11.861071 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.860525    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:11.861071 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.860546    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:11.861071 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.860558    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:11.861071 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:11.860620    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:15.860600768 +0000 UTC m=+10.229253681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:12.181171 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:12.180608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:12.181171 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:12.180769    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:13.181222 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:13.181155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:13.181595 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:13.181298    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:13.181686 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:13.181653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:13.181771 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:13.181750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:14.183674 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:14.182215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:14.183674 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:14.182373    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:15.181428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:15.181398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:15.181612 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.181510    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:15.181940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:15.181904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:15.182045 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.182000    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:15.694682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:15.694634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:15.695171 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.694816    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:15.695171 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.694905    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:23.694881618 +0000 UTC m=+18.063534513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:15.795651 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:15.795609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:15.795828 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.795745    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:15.795828 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.795809    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:23.795790961 +0000 UTC m=+18.164443918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:15.896226 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:15.896182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:15.896444 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.896422    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:15.896523 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.896454    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:15.896523 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.896468    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:15.896609 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:15.896530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:23.896511073 +0000 UTC m=+18.265163985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:16.182550 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:16.182473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:16.182691 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:16.182600    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:17.181328 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:17.181299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:17.181715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:17.181346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:17.181715 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:17.181414    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:17.181715 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:17.181553    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:18.185449 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:18.185416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:18.185906 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:18.185549    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:19.181426 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:19.181381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:19.181615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:19.181396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:19.181615 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:19.181522    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:19.181615 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:19.181584    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:20.180951 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:20.180910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:20.181370 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:20.181084    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:21.181548 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:21.181513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:21.182009 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:21.181580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:21.182009 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:21.181681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:21.182255 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:21.182217    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:22.180682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:22.180642    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:22.180850 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:22.180788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:23.181190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:23.181154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:23.181698 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.181284    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:23.181698 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:23.181163    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:23.181771 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.181711    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:23.757715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:23.757682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:23.757857 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.757798    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:23.757857 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.757847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.757834717 +0000 UTC m=+34.126487611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:23.858734 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:23.858700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:23.858882 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.858846    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:23.858933 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.858909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.858894332 +0000 UTC m=+34.227547227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:23.959798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:23.959761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:23.959980 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.959946    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:23.959980 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.959971    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:23.960096 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.959985    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:23.960096 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:23.960041    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:33:39.960027611 +0000 UTC m=+34.328680504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:24.181043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:24.181007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:24.181241 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:24.181148    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:25.180730 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:25.180687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:25.180910 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:25.180687    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:25.180910 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:25.180842    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:25.180910 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:25.180880    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:26.181739 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:26.181695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:26.182161 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:26.181841    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:27.180600 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:27.180561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:27.180772 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:27.180561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:27.180772 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:27.180690    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:27.180772 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:27.180756    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:28.184197 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.181461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:28.184197 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:28.181795    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:28.278684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.278491    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7" event={"ID":"163fe8a6-6224-4bb1-9a53-e2f67f35e234","Type":"ContainerStarted","Data":"e44d39f26dd96f1f3b5179b672ee02773b1ad267e949743c7e42e86d465154dd"}
Jul 28 15:33:28.279890 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.279862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"b77867ab0e7dcbdc6bcb54ea0b27bbaae1a52e146b8debbe9d559d3e1da70701"}
Jul 28 15:33:28.280019 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.279927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"9e90220ea96401fde1612e3bd722d867e47c6dd5726dc2ae4a41bc143d5430a2"}
Jul 28 15:33:28.281203 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.281182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerStarted","Data":"d9993c8cb4a6fbb1383ff07eb19e09a662c2c906a3988a3bafd8f95b46b1956e"}
Jul 28 15:33:28.282683 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.282663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbx6n" event={"ID":"ff52c32e-08f8-4417-bda9-9eddd9a13efc","Type":"ContainerStarted","Data":"c8bc12e2459ee0e14cdf6f684b43ab43b40f0a69c57ed50626f33871216420c8"}
Jul 28 15:33:28.284163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.284142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z7bcq" event={"ID":"119e856f-abfc-42cd-9d2c-052ed52df947","Type":"ContainerStarted","Data":"43484569d9f089e933823dd0aa35de95138d8637eb1e0582f04ed53859038ccc"}
Jul 28 15:33:28.285487 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.285465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m6mgg" event={"ID":"adee8374-4c30-41b2-a493-6274dfa19bc5","Type":"ContainerStarted","Data":"5be2a6c984ef8ecfe4b264035c2d81abe8b1466a349edda86722170d67f1b6ab"}
Jul 28 15:33:28.286740 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.286717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drbzm" event={"ID":"0dfe30ab-2aa2-416f-a1d8-f72f0767044a","Type":"ContainerStarted","Data":"e945795e7580fd72bdb1301c93b4acfe8d59d2f09af0c730f9c0f5b4ab094318"}
Jul 28 15:33:28.288410 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.288393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qn2n4" event={"ID":"aeaab55a-2ceb-4543-a282-3a3ebebcb0af","Type":"ContainerStarted","Data":"3d117554b195626b257de2d7adcbac12a63716eeebcefd57d95536935fcd07ea"}
Jul 28 15:33:28.378384 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.378338    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-drbzm" podStartSLOduration=3.334289792 podStartE2EDuration="22.378322836s" totalImagesPullingTime="19.044033044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.378239333 +0000 UTC m=+22.746892249" watchObservedRunningTime="2026-07-28 15:33:28.378322836 +0000 UTC m=+22.746975748"
Jul 28 15:33:28.443236 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.443193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qn2n4" podStartSLOduration=7.735753216 podStartE2EDuration="22.443173891s" totalImagesPullingTime="14.707420675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.402391065 +0000 UTC m=+22.771043982" watchObservedRunningTime="2026-07-28 15:33:28.443173891 +0000 UTC m=+22.811826811"
Jul 28 15:33:28.443664 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.443625    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m6mgg" podStartSLOduration=3.447179808 podStartE2EDuration="22.4436149s" totalImagesPullingTime="18.996435092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.44299211 +0000 UTC m=+22.811645026" watchObservedRunningTime="2026-07-28 15:33:28.4436149 +0000 UTC m=+22.812267816"
Jul 28 15:33:28.517413 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.517299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z7bcq" podStartSLOduration=3.5778701269999997 podStartE2EDuration="22.517282993s" totalImagesPullingTime="18.939412866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.480624874 +0000 UTC m=+22.849277790" watchObservedRunningTime="2026-07-28 15:33:28.517282993 +0000 UTC m=+22.885935909"
Jul 28 15:33:28.517716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.517691    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dbx6n" podStartSLOduration=11.793128421 podStartE2EDuration="22.51768161s" totalImagesPullingTime="10.724553189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:28.517212848 +0000 UTC m=+22.885865768" watchObservedRunningTime="2026-07-28 15:33:28.51768161 +0000 UTC m=+22.886334525"
Jul 28 15:33:28.607421 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.607397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:28.719114 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.719071    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:28.811818 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:28.811787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:29.181345 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.181279    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:29.181345 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.181310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:29.181572 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:29.181386    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:29.181572 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:29.181439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:29.293172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.293119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"d0c778a1d12fcb9159f1c11f2e32b518926984404deb8bb72592a03940d90ab7"}
Jul 28 15:33:29.293868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.293223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"9dd38276f828d927b0bd6e064ab4badd6a8e445d4d5c2d9214dff004bd5fa220"}
Jul 28 15:33:29.293868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.293275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"1735df7cd5dfdfad0cb2bae1333c14f00ea3553caf61a288287f420380b8e335"}
Jul 28 15:33:29.293868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.293324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"5b2ffdad8a17d6d7c24938fb0d4d596e291956fcfb41d489b0f5e7c807aeaa19"}
Jul 28 15:33:29.294645 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.294621    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="d9993c8cb4a6fbb1383ff07eb19e09a662c2c906a3988a3bafd8f95b46b1956e" exitCode=0
Jul 28 15:33:29.294760 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.294732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"d9993c8cb4a6fbb1383ff07eb19e09a662c2c906a3988a3bafd8f95b46b1956e"}
Jul 28 15:33:29.476630 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:29.476598    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 15:33:30.082140 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.081948    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T15:33:29.476615993Z","UUID":"393980c8-f47c-425b-ae84-06fe0cff4ef0","Handler":null,"Name":"","Endpoint":""}
Jul 28 15:33:30.084303 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.083989    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 15:33:30.084303 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.084032    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 15:33:30.181176 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.181116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:30.181333 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:30.181288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:30.298579 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.298545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rcwf8" event={"ID":"61cf79f4-d5f4-49fb-991a-1618cb42a3f6","Type":"ContainerStarted","Data":"ef96ed67ea2897b3a82eec29606a3f6cbcfd31151fffbe28d9009b2dad1bea7e"}
Jul 28 15:33:30.301508 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.301476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7" event={"ID":"163fe8a6-6224-4bb1-9a53-e2f67f35e234","Type":"ContainerStarted","Data":"7a04a5be967b2b6511176890c1417e333d0b502d96783212f2bdc923673a2051"}
Jul 28 15:33:30.322266 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.322218    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rcwf8" podStartSLOduration=5.328216305 podStartE2EDuration="24.322197815s" totalImagesPullingTime="18.99398151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:30.322108417 +0000 UTC m=+24.690761334" watchObservedRunningTime="2026-07-28 15:33:30.322197815 +0000 UTC m=+24.690850750"
Jul 28 15:33:30.456756 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.456721    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:30.589994 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:30.589930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:31.180769 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.180732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:31.180965 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:31.180889    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:31.180965 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.180946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:31.181148 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:31.181038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:31.297497 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.297462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:31.307163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.306679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7" event={"ID":"163fe8a6-6224-4bb1-9a53-e2f67f35e234","Type":"ContainerStarted","Data":"1beebaabd4d6d09e77ebb2fb9e0517c4d4feefcaf05c9926a081d2461d03ed9f"}
Jul 28 15:33:31.310412 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.310383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"81b308d2c7dd011ba2b0e78d0de713307ac176e2db13b5b3e836c281bf572a76"}
Jul 28 15:33:31.333643 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.333590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcjq7" podStartSLOduration=3.948108323 podStartE2EDuration="25.333519118s" totalImagesPullingTime="21.385410795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:31.332926111 +0000 UTC m=+25.701579027" watchObservedRunningTime="2026-07-28 15:33:31.333519118 +0000 UTC m=+25.702172036"
Jul 28 15:33:31.396147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.396097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:31.794380 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.794187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:31.794921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:31.794895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:32.181584 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:32.181550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:32.181770 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:32.181675    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:32.312904 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:32.312872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:32.313340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:32.313320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbx6n"
Jul 28 15:33:32.418730 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:32.418699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:33.180525 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:33.180467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:33.180713 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:33.180467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:33.180713 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:33.180624    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:33.180713 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:33.180685    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:34.181751 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.181567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:34.182451 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:34.181854    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:34.320928 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.320892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" event={"ID":"1eb13ef3-41ab-45ce-9a59-fc57643325f7","Type":"ContainerStarted","Data":"635d1fa3d4f95bf3481880b75c99b7bc13eea3e24891d7f956f3630c16d257dd"}
Jul 28 15:33:34.321246 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.321224    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:34.322891 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.322861    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="cf0490efee750a996d41644871314d44ba7f7f06a5eb62ef14a5f2657a8e0f48" exitCode=0
Jul 28 15:33:34.322997 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.322910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"cf0490efee750a996d41644871314d44ba7f7f06a5eb62ef14a5f2657a8e0f48"}
Jul 28 15:33:34.337893 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.337869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:34.364715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:34.364663    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94" podStartSLOduration=9.326967487 podStartE2EDuration="28.364649584s" totalImagesPullingTime="19.037682097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:34.361631621 +0000 UTC m=+28.730284547" watchObservedRunningTime="2026-07-28 15:33:34.364649584 +0000 UTC m=+28.733302500"
Jul 28 15:33:35.180615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.180583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:35.180853 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:35.180689    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:35.180853 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.180741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:35.180853 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:35.180825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:35.326383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.326356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:35.326775 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.326436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:35.341913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.341887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:33:35.757404 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.757372    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:35.861279 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.861243    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:35.930789 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.930757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bnw82"]
Jul 28 15:33:35.930941 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.930857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:35.930995 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:35.930939    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:35.953761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.953732    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krkl4"]
Jul 28 15:33:35.953912 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.953843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:35.953960 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:35.953923    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:35.962758 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.962730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jstzl"]
Jul 28 15:33:35.962868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:35.962817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:35.962921 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:35.962892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:36.227215 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.227177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:36.330408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.330376    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="addf304c4b148872ca8afccff3bc3e6e9d11ac0119d3787e346b0f89070ce4de" exitCode=0
Jul 28 15:33:36.330839 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.330464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"addf304c4b148872ca8afccff3bc3e6e9d11ac0119d3787e346b0f89070ce4de"}
Jul 28 15:33:36.333288 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.333266    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:36.504580 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.504504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:36.601201 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:36.601170    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:37.140429 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:37.140400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:37.180682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:37.180611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:37.180838 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:37.180727    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:37.269785 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:37.269574    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:37.377225 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:37.377189    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.162787 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.162745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.180970 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.180935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:38.181168 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:38.181076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:38.181168 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.181119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:38.181292 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:38.181231    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:38.261703 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.261662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:38.336939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.336909    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="1ac63215c828910a6ac9e3c5116aa8399ac3708d6f80a1b9bbf5ee286e18cfcb" exitCode=0
Jul 28 15:33:38.337091 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.336961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"1ac63215c828910a6ac9e3c5116aa8399ac3708d6f80a1b9bbf5ee286e18cfcb"}
Jul 28 15:33:38.369803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:38.369771    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:39.181565 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:39.181486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:39.182059 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.181609    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:39.743738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:39.743702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:39.778885 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:39.778844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:39.779068 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.779017    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:39.779117 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.779082    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret podName:3734d978-9bc8-4025-99ee-f847754332e5 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.779068851 +0000 UTC m=+66.147721763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret") pod "global-pull-secret-syncer-jstzl" (UID: "3734d978-9bc8-4025-99ee-f847754332e5") : object "kube-system"/"original-pull-secret" not registered
Jul 28 15:33:39.879833 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:39.879802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:39.879990 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.879895    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:39.879990 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.879944    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs podName:586a1d1d-6a0d-4cf6-9df0-41ee96f329a2 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.87992992 +0000 UTC m=+66.248582814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs") pod "network-metrics-daemon-krkl4" (UID: "586a1d1d-6a0d-4cf6-9df0-41ee96f329a2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 15:33:39.980984 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:39.980944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:39.981181 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.981100    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 15:33:39.981181 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.981119    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 15:33:39.981181 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.981150    2580 projected.go:196] Error preparing data for projected volume kube-api-access-c2hzh for pod openshift-network-diagnostics/network-check-target-bnw82: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:39.981302 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:39.981208    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh podName:3fa94af5-82f1-42a5-9a55-3de4251112e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:34:11.981196171 +0000 UTC m=+66.349849065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c2hzh" (UniqueName: "kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh") pod "network-check-target-bnw82" (UID: "3fa94af5-82f1-42a5-9a55-3de4251112e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 15:33:40.012362 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:40.012280    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z7bcq_119e856f-abfc-42cd-9d2c-052ed52df947/dns-node-resolver/0.log"
Jul 28 15:33:40.180570 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:40.180510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:40.180754 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:40.180642    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:40.180754 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:40.180678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:40.180754 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:40.180744    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:40.813331 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:40.813306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn2n4_aeaab55a-2ceb-4543-a282-3a3ebebcb0af/node-ca/0.log"
Jul 28 15:33:41.181544 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:41.181513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:41.181729 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:41.181651    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:41.850292 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:41.850228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:41.956794 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:41.956744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.044506 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.044466    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.181941 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.181554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:42.181941 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.181578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:42.181941 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:42.181700    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:42.181941 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:42.181849    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:42.269170 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.269122    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.371992 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.371959    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.472571 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.472533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.569761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.569722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.739870 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.739785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:42.883366 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:42.883339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:43.180794 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:43.180760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:43.180980 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:43.180878    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:43.352903 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:43.352865    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="cc87776cf8b63dabed32515eb5020d3f5d864210fb2241d24d3727053ac18d41" exitCode=0
Jul 28 15:33:43.353073 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:43.352908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"cc87776cf8b63dabed32515eb5020d3f5d864210fb2241d24d3727053ac18d41"}
Jul 28 15:33:43.754142 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:43.754037    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:44.108053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.108023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:44.180800 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.180765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:44.180980 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:44.180869    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:44.180980 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.180934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:44.181079 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:44.181058    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:44.328024 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.327987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:44.358043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.358011    2580 generic.go:356] "Generic (PLEG): container finished" podID="3702206b-0647-4ea7-8cef-4fa6b7fa1fdd" containerID="6ea4636161486bcc3ce832a371ff50c981b1a85a937d5a83ee9f731f5c227824" exitCode=0
Jul 28 15:33:44.358239 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.358067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerDied","Data":"6ea4636161486bcc3ce832a371ff50c981b1a85a937d5a83ee9f731f5c227824"}
Jul 28 15:33:44.518739 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.518711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:44.937744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:44.937714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:45.181494 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.181461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:45.181852 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:45.181581    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:45.316796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.316717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:45.363626 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.363594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" event={"ID":"3702206b-0647-4ea7-8cef-4fa6b7fa1fdd","Type":"ContainerStarted","Data":"e7aef3077da9247a2c5378e4808320d696d5f62ba6510344f71bc8ee674ca99b"}
Jul 28 15:33:45.392435 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.392386    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sxwcf" podStartSLOduration=10.132499517 podStartE2EDuration="39.392369849s" totalImagesPullingTime="29.259870332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:33:45.391929008 +0000 UTC m=+39.760581937" watchObservedRunningTime="2026-07-28 15:33:45.392369849 +0000 UTC m=+39.761022764"
Jul 28 15:33:45.635987 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.635960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:45.736437 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.736388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:45.898359 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:45.898279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:46.181925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.181713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:46.182284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.181789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:46.182284 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:46.181972    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:46.182284 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:46.182027    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:46.205605 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.205573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:46.301962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.301917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:46.404672 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.404645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:46.502737 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.502648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:46.928842 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:46.928808    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:47.181111 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:47.181027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:47.181297 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:47.181157    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:47.505742 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:47.505665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:47.902307 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:47.902272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:48.117687 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.117651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:48.180679 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.180601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:48.180822 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:48.180724    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:48.180822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.180779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:48.180923 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:48.180902    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:48.352619 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.352583    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:48.710312 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.710272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:48.902319 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:48.902288    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.180961 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:49.180931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:49.181178 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:49.181030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:49.308557 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:49.308525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.408815 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:49.408775    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:49.706290 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:49.706255    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:50.180744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:50.180714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:50.181153 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:50.180825    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:50.181153 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:50.180901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:50.181153 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:50.180998    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:50.307578 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:50.307534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:50.736925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:50.736892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.124206 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:51.124173    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.181390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:51.181357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:51.181725 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:51.181456    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:51.511890 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:51.511793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.821399 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:51.821320    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:51.930901 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:51.930865    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:52.101925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:52.101889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:52.181517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:52.181485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:52.181517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:52.181500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:52.181919 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:52.181593    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:52.181919 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:52.181739    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:52.513234 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:52.513145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:52.738443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:52.738414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:53.181491 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:53.181438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:53.181662 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:53.181561    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:53.371337 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:53.371294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:53.471039 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:53.470960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:53.655402 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:53.655363    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.040846 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.040811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.145147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.145094    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.181335 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.181307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:54.181466 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:54.181427    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:54.181509 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.181490    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:54.181626 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:54.181605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:54.441013 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.440983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.545748 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.545714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.650255 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.650224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.733998 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.733918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:54.849145 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:54.849092    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:55.049885 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:55.049806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:55.180863 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:55.180824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:55.181024 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:55.180933    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:55.894394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:55.894358    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:56.184574 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:56.184491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:56.184834 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:56.184806    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:56.184924 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:56.184562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:56.184924 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:56.184905    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:56.463013 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:56.462911    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:56.605211 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:56.605181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:57.180774 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:57.180736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:57.181183 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:57.180843    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:33:58.180887 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:58.180855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:33:58.181367 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:58.180958    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:33:58.181367 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:58.181033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:33:58.181367 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:58.181145    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:33:58.246538 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:58.246505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:58.347331 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:58.347296    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:33:59.181504 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:33:59.181476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:33:59.181875 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:33:59.181576    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:34:00.180796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:00.180761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:34:00.180984 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:00.180863    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jstzl" podUID="3734d978-9bc8-4025-99ee-f847754332e5"
Jul 28 15:34:00.180984 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:00.180953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:34:00.181098 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:00.181079    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krkl4" podUID="586a1d1d-6a0d-4cf6-9df0-41ee96f329a2"
Jul 28 15:34:01.180251 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.180215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:01.181104 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.181085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:01.181239 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:01.181218    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bnw82" podUID="3fa94af5-82f1-42a5-9a55-3de4251112e1"
Jul 28 15:34:01.281706 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.281670    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:01.381477 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.381432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:01.479393 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.479274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.000001 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:01.999972    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeReady"
Jul 28 15:34:02.000187 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.000105    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 15:34:02.049140 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.049093    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fkh4t"]
Jul 28 15:34:02.050778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.050763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.052351 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.052328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jhq"]
Jul 28 15:34:02.053293 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.053272    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2rz82\""
Jul 28 15:34:02.053803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.053785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 15:34:02.053902 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.053805    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 15:34:02.053902 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.053790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 15:34:02.054193 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.054174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.056780 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.056745    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 15:34:02.057028 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.057009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hgr8\""
Jul 28 15:34:02.057272 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.057257    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 15:34:02.057586 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.057568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 15:34:02.057840 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.057823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 15:34:02.065398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.065378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkh4t"]
Jul 28 15:34:02.067555 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.067534    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jhq"]
Jul 28 15:34:02.150097 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.150057    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fcj9m"]
Jul 28 15:34:02.152397 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.152382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.154989 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.154957    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 15:34:02.155147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.155103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 15:34:02.155147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.155106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ks94b\""
Jul 28 15:34:02.163175 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.163150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcj9m"]
Jul 28 15:34:02.181475 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.181449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:34:02.181894 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.181638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:34:02.186731 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.186697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:34:02.186850 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.186697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:34:02.186850 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.186803    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hctgx\""
Jul 28 15:34:02.232104 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.232276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-cert\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.232276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.232276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqr4\" (UniqueName: \"kubernetes.io/projected/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-api-access-sjqr4\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.232276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trclk\" (UniqueName: \"kubernetes.io/projected/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-kube-api-access-trclk\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.232276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-crio-socket\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.232460 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.232305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-data-volume\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.282307 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.282185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.332968 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.332933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.332980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-cert\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.332998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqr4\" (UniqueName: \"kubernetes.io/projected/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-api-access-sjqr4\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trclk\" (UniqueName: \"kubernetes.io/projected/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-kube-api-access-trclk\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-crio-socket\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-config-volume\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.333482 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-data-volume\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333482 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7rx\" (UniqueName: \"kubernetes.io/projected/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-kube-api-access-rk7rx\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.333482 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-metrics-tls\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.333482 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-tmp-dir\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.333482 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-crio-socket\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-data-volume\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.333729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.333599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.336949 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.336919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.337068 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.337027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-cert\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.342558 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.342526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trclk\" (UniqueName: \"kubernetes.io/projected/12bc6b40-3bc3-4c72-8a9a-2320ec05806e-kube-api-access-trclk\") pod \"ingress-canary-fkh4t\" (UID: \"12bc6b40-3bc3-4c72-8a9a-2320ec05806e\") " pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.343393 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.343371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqr4\" (UniqueName: \"kubernetes.io/projected/03d46c54-4f83-41a9-9d99-e2f88aec3f4a-kube-api-access-sjqr4\") pod \"insights-runtime-extractor-h6jhq\" (UID: \"03d46c54-4f83-41a9-9d99-e2f88aec3f4a\") " pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.362200 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.362157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkh4t"
Jul 28 15:34:02.367076 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.367054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6jhq"
Jul 28 15:34:02.435089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.433766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7rx\" (UniqueName: \"kubernetes.io/projected/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-kube-api-access-rk7rx\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.435089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.433815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-metrics-tls\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.435089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.433838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-tmp-dir\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.435089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.433926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-config-volume\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.435089 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.434496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-config-volume\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.436825 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.435444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-tmp-dir\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.440615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.440558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-metrics-tls\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.449411 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.449383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7rx\" (UniqueName: \"kubernetes.io/projected/3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b-kube-api-access-rk7rx\") pod \"dns-default-fcj9m\" (UID: \"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b\") " pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.463529 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.462305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:02.527456 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.527368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6jhq"]
Jul 28 15:34:02.535819 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:02.535784    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d46c54_4f83_41a9_9d99_e2f88aec3f4a.slice/crio-0f221d9cf3754127e0a58b564480ae10f3407edba2c763b48cc9b1e2acb63ef3 WatchSource:0}: Error finding container 0f221d9cf3754127e0a58b564480ae10f3407edba2c763b48cc9b1e2acb63ef3: Status 404 returned error can't find the container with id 0f221d9cf3754127e0a58b564480ae10f3407edba2c763b48cc9b1e2acb63ef3
Jul 28 15:34:02.548818 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.548777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkh4t"]
Jul 28 15:34:02.552327 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:02.552296    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bc6b40_3bc3_4c72_8a9a_2320ec05806e.slice/crio-e4913eaa972809565bd020e7e8c5f1895f1490e0a31dbf2ba806d742507e8192 WatchSource:0}: Error finding container e4913eaa972809565bd020e7e8c5f1895f1490e0a31dbf2ba806d742507e8192: Status 404 returned error can't find the container with id e4913eaa972809565bd020e7e8c5f1895f1490e0a31dbf2ba806d742507e8192
Jul 28 15:34:02.596290 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.596260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fcj9m"]
Jul 28 15:34:02.599950 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:02.599923    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6178ec_7634_41cf_8bc4_0d5eec6a7d2b.slice/crio-84206618a62fba7335c19f3b63719f95c7b2d0a86388d82445086fd14be5c109 WatchSource:0}: Error finding container 84206618a62fba7335c19f3b63719f95c7b2d0a86388d82445086fd14be5c109: Status 404 returned error can't find the container with id 84206618a62fba7335c19f3b63719f95c7b2d0a86388d82445086fd14be5c109
Jul 28 15:34:02.679469 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.679433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:02.743508 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:02.743476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.180971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.180931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:03.185190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.184731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djz96\""
Jul 28 15:34:03.185190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.184994    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:34:03.188869 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.187275    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:34:03.270771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.270674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.287706 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.286709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:03.289920 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.289089    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.292593 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.292308    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dpkp2\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293005    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293570    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.293864    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 15:34:03.294284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.294101    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 15:34:03.304543 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.304460    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:03.369450 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.369383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:03.408247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.408206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcj9m" event={"ID":"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b","Type":"ContainerStarted","Data":"84206618a62fba7335c19f3b63719f95c7b2d0a86388d82445086fd14be5c109"}
Jul 28 15:34:03.411747 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.411673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jhq" event={"ID":"03d46c54-4f83-41a9-9d99-e2f88aec3f4a","Type":"ContainerStarted","Data":"0f221d9cf3754127e0a58b564480ae10f3407edba2c763b48cc9b1e2acb63ef3"}
Jul 28 15:34:03.414272 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.414185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkh4t" event={"ID":"12bc6b40-3bc3-4c72-8a9a-2320ec05806e","Type":"ContainerStarted","Data":"e4913eaa972809565bd020e7e8c5f1895f1490e0a31dbf2ba806d742507e8192"}
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.440812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.440860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.440886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckql\" (UniqueName: \"kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.440965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.440996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.441209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.441030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.541874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckql\" (UniqueName: \"kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542767 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.542743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.542914 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.542890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.543461 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.543382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.549854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.549732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.549854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.549796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.554091 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.554048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckql\" (UniqueName: \"kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql\") pod \"console-7c9bd44d8b-vs7xc\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") " pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.602811 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.602775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:03.784271 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:03.784241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:03.789183 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:03.789155    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec385968_1fcf_44df_a9f3_91d2eb414e00.slice/crio-3432c52ef6149143088d0e881699cd98ed273b6ef144973fec8fa094d0e22f63 WatchSource:0}: Error finding container 3432c52ef6149143088d0e881699cd98ed273b6ef144973fec8fa094d0e22f63: Status 404 returned error can't find the container with id 3432c52ef6149143088d0e881699cd98ed273b6ef144973fec8fa094d0e22f63
Jul 28 15:34:04.418686 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:04.418652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd44d8b-vs7xc" event={"ID":"ec385968-1fcf-44df-a9f3-91d2eb414e00","Type":"ContainerStarted","Data":"3432c52ef6149143088d0e881699cd98ed273b6ef144973fec8fa094d0e22f63"}
Jul 28 15:34:05.655269 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.655236    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:34:05.666796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.666765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.669872 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.669840    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:34:05.670382 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.670333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.675969 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.675939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 15:34:05.761610 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmgh\" (UniqueName: \"kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.761972 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.761810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.765904 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.765876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:05.862217 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmgh\" (UniqueName: \"kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.862684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.862428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.863093 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.863066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.863238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.863117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.863308 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.863234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.863571 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.863550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.866244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.866197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.866544 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.866519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.872592 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.872557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmgh\" (UniqueName: \"kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh\") pod \"console-769b9fbd8c-4nxqw\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") " pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:05.979249 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:05.979147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:07.346097 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.346064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dc94"
Jul 28 15:34:07.373949 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.373917    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.471644 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.471613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.548512 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.548480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:07.551352 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.551319    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"]
Jul 28 15:34:07.566449 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.566422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"]
Jul 28 15:34:07.566600 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.566561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:07.569395 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.569174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 15:34:07.569395 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.569187    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7jshn\""
Jul 28 15:34:07.578927 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.578899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t2ghz\" (UID: \"820c66c6-0ee9-41f3-9090-a8e676b3a42a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:07.679885 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.679803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t2ghz\" (UID: \"820c66c6-0ee9-41f3-9090-a8e676b3a42a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:07.680028 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:07.679915    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:07.680028 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:07.679967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates podName:820c66c6-0ee9-41f3-9090-a8e676b3a42a nodeName:}" failed. No retries permitted until 2026-07-28 15:34:08.17995245 +0000 UTC m=+62.548605356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-t2ghz" (UID: "820c66c6-0ee9-41f3-9090-a8e676b3a42a") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 15:34:07.768496 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:07.768464    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:08.183205 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:08.183166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t2ghz\" (UID: \"820c66c6-0ee9-41f3-9090-a8e676b3a42a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:08.185930 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:08.185903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/820c66c6-0ee9-41f3-9090-a8e676b3a42a-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t2ghz\" (UID: \"820c66c6-0ee9-41f3-9090-a8e676b3a42a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:08.478875 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:08.478787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:09.680637 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:09.680592    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:09.783282 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:09.783247    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:09.986576 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:09.986500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:34:10.027573 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.027543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"]
Jul 28 15:34:10.438332 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.438284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jhq" event={"ID":"03d46c54-4f83-41a9-9d99-e2f88aec3f4a","Type":"ContainerStarted","Data":"cf3b311ca78a23d2310014b92e4deb7c01a1ac141bac153632911bbe704b72af"}
Jul 28 15:34:10.440453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.440384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkh4t" event={"ID":"12bc6b40-3bc3-4c72-8a9a-2320ec05806e","Type":"ContainerStarted","Data":"9639bb4eaac59af4858e6a09f2ce9f0c7b10137fb7585aa9b63d82147b1061f9"}
Jul 28 15:34:10.442918 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.442891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcj9m" event={"ID":"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b","Type":"ContainerStarted","Data":"a049b03afd18185bc7a25e65625bd8037d5554aa5d36347a2987e6104d99013b"}
Jul 28 15:34:10.443038 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.442942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fcj9m" event={"ID":"3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b","Type":"ContainerStarted","Data":"c3f49285be20e592c1473f52a94da2883173acb851a01824f0a5900a1869fa9d"}
Jul 28 15:34:10.443536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.443449    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:10.446258 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.446191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz" event={"ID":"820c66c6-0ee9-41f3-9090-a8e676b3a42a","Type":"ContainerStarted","Data":"6c9bdcd0183626f1d8659dcb8406496103fef9a1251ddb3a4a01d71fe08ef76f"}
Jul 28 15:34:10.448356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.448279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd44d8b-vs7xc" event={"ID":"ec385968-1fcf-44df-a9f3-91d2eb414e00","Type":"ContainerStarted","Data":"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"}
Jul 28 15:34:10.449939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.449871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769b9fbd8c-4nxqw" event={"ID":"4c9f8714-2aae-4a52-b6ef-04654f186af3","Type":"ContainerStarted","Data":"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"}
Jul 28 15:34:10.449939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.449909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769b9fbd8c-4nxqw" event={"ID":"4c9f8714-2aae-4a52-b6ef-04654f186af3","Type":"ContainerStarted","Data":"9f9d09620a0f32d208590fb44b46a6f7ede8a309428fda26f3a1551b1bca28b8"}
Jul 28 15:34:10.461117 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.461080    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fkh4t" podStartSLOduration=1.17856146 podStartE2EDuration="8.461064649s" totalImagesPullingTime="7.282503189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.460322775 +0000 UTC m=+64.828975691" watchObservedRunningTime="2026-07-28 15:34:10.461064649 +0000 UTC m=+64.829717568"
Jul 28 15:34:10.506251 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.506190    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fcj9m" podStartSLOduration=1.293806225 podStartE2EDuration="8.506169395s" totalImagesPullingTime="7.21236317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.505868218 +0000 UTC m=+64.874521135" watchObservedRunningTime="2026-07-28 15:34:10.506169395 +0000 UTC m=+64.874822309"
Jul 28 15:34:10.506698 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.506658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-769b9fbd8c-4nxqw" podStartSLOduration=5.506646305 podStartE2EDuration="5.506646305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.480812976 +0000 UTC m=+64.849465894" watchObservedRunningTime="2026-07-28 15:34:10.506646305 +0000 UTC m=+64.875299223"
Jul 28 15:34:10.528645 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:10.528588    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9bd44d8b-vs7xc" podStartSLOduration=1.457774281 podStartE2EDuration="7.52856272s" totalImagesPullingTime="6.070788439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:10.527680555 +0000 UTC m=+64.896333464" watchObservedRunningTime="2026-07-28 15:34:10.52856272 +0000 UTC m=+64.897215637"
Jul 28 15:34:11.396813 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.396772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.455968 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.455933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz" event={"ID":"820c66c6-0ee9-41f3-9090-a8e676b3a42a","Type":"ContainerStarted","Data":"7d9c1a1ea2c31f52553dffb3d3d0dde07d960684153c13a4082687dac4039f1a"}
Jul 28 15:34:11.473096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.473041    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz" podStartSLOduration=3.247290583 podStartE2EDuration="4.473025994s" totalImagesPullingTime="1.225735411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:11.472443487 +0000 UTC m=+65.841096404" watchObservedRunningTime="2026-07-28 15:34:11.473025994 +0000 UTC m=+65.841678910"
Jul 28 15:34:11.479056 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.479029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.553644 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.553562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.810199 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.810089    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.817167 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.817112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:34:11.819996 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.819965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 15:34:11.829878 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.829852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3734d978-9bc8-4025-99ee-f847754332e5-original-pull-secret\") pod \"global-pull-secret-syncer-jstzl\" (UID: \"3734d978-9bc8-4025-99ee-f847754332e5\") " pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:34:11.872092 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.872062    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:11.917580 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.917538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:34:11.920726 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.920695    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 15:34:11.930684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:11.930657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586a1d1d-6a0d-4cf6-9df0-41ee96f329a2-metrics-certs\") pod \"network-metrics-daemon-krkl4\" (UID: \"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2\") " pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:34:12.017971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.017927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:12.021537 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.021516    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 15:34:12.031853 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.031822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 15:34:12.042340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.042282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hzh\" (UniqueName: \"kubernetes.io/projected/3fa94af5-82f1-42a5-9a55-3de4251112e1-kube-api-access-c2hzh\") pod \"network-check-target-bnw82\" (UID: \"3fa94af5-82f1-42a5-9a55-3de4251112e1\") " pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:12.094078 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.094043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jstzl"
Jul 28 15:34:12.101431 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.101398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hctgx\""
Jul 28 15:34:12.108438 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.108418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krkl4"
Jul 28 15:34:12.199183 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.199156    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-djz96\""
Jul 28 15:34:12.206659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.206631    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:12.381632 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.381555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.458467 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.458430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.460226 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.460172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:12.466020 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.465994    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t2ghz"
Jul 28 15:34:12.598467 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.598435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.634791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.634697    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"]
Jul 28 15:34:12.699396 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.699368    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.742394 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.742358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"]
Jul 28 15:34:12.742558 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.742350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.745476 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 15:34:12.745618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745484    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b67vk\""
Jul 28 15:34:12.745618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 15:34:12.745618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 15:34:12.745618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745591    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 15:34:12.745802 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.745769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 15:34:12.798832 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.798796    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:12.824909 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.824880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.825055 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.824917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.825055 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.824954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bxr\" (UniqueName: \"kubernetes.io/projected/616bd69c-c54b-4977-a225-afbbff19ec3c-kube-api-access-k7bxr\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.825055 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.824990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616bd69c-c54b-4977-a225-afbbff19ec3c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.925813 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.925726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.925813 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.925774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.926031 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.925934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bxr\" (UniqueName: \"kubernetes.io/projected/616bd69c-c54b-4977-a225-afbbff19ec3c-kube-api-access-k7bxr\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.926031 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.926014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616bd69c-c54b-4977-a225-afbbff19ec3c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.926676 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.926651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616bd69c-c54b-4977-a225-afbbff19ec3c-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.928390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.928364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.928390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.928370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616bd69c-c54b-4977-a225-afbbff19ec3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:12.934971 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:12.934951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bxr\" (UniqueName: \"kubernetes.io/projected/616bd69c-c54b-4977-a225-afbbff19ec3c-kube-api-access-k7bxr\") pod \"prometheus-operator-84d69f794d-q9g5g\" (UID: \"616bd69c-c54b-4977-a225-afbbff19ec3c\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:13.052460 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.052419    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"
Jul 28 15:34:13.466486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.466448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:13.566741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.566708    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:13.603713 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.603671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:13.603890 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.603740    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:13.608395 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.608374    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:13.958428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:13.958398    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bnw82"]
Jul 28 15:34:13.962389 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:13.962361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa94af5_82f1_42a5_9a55_3de4251112e1.slice/crio-22b0f629e00e0acde9f6e5e197f70c052d4e87a6ed4b52a8f62ea983ac799bc3 WatchSource:0}: Error finding container 22b0f629e00e0acde9f6e5e197f70c052d4e87a6ed4b52a8f62ea983ac799bc3: Status 404 returned error can't find the container with id 22b0f629e00e0acde9f6e5e197f70c052d4e87a6ed4b52a8f62ea983ac799bc3
Jul 28 15:34:14.186766 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.186738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krkl4"]
Jul 28 15:34:14.190105 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:14.190076    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586a1d1d_6a0d_4cf6_9df0_41ee96f329a2.slice/crio-218d91a17d7e1bcbc6d3b1754a1473fc8d530d921fbaa850c515e5e2aee5eab2 WatchSource:0}: Error finding container 218d91a17d7e1bcbc6d3b1754a1473fc8d530d921fbaa850c515e5e2aee5eab2: Status 404 returned error can't find the container with id 218d91a17d7e1bcbc6d3b1754a1473fc8d530d921fbaa850c515e5e2aee5eab2
Jul 28 15:34:14.191093 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.191013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jstzl"]
Jul 28 15:34:14.192264 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.192216    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-q9g5g"]
Jul 28 15:34:14.193656 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:14.193633    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616bd69c_c54b_4977_a225_afbbff19ec3c.slice/crio-3b5236feb7d91f93fa09f0d7bf56bfe6c3a41e1ffebab406854004fd5de0d8f7 WatchSource:0}: Error finding container 3b5236feb7d91f93fa09f0d7bf56bfe6c3a41e1ffebab406854004fd5de0d8f7: Status 404 returned error can't find the container with id 3b5236feb7d91f93fa09f0d7bf56bfe6c3a41e1ffebab406854004fd5de0d8f7
Jul 28 15:34:14.195191 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:14.195165    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3734d978_9bc8_4025_99ee_f847754332e5.slice/crio-d663f7e6fcbed13d3345330aa1c16878e0fc360c149b3a683ae8d869916192d9 WatchSource:0}: Error finding container d663f7e6fcbed13d3345330aa1c16878e0fc360c149b3a683ae8d869916192d9: Status 404 returned error can't find the container with id d663f7e6fcbed13d3345330aa1c16878e0fc360c149b3a683ae8d869916192d9
Jul 28 15:34:14.470978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.470875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jstzl" event={"ID":"3734d978-9bc8-4025-99ee-f847754332e5","Type":"ContainerStarted","Data":"d663f7e6fcbed13d3345330aa1c16878e0fc360c149b3a683ae8d869916192d9"}
Jul 28 15:34:14.474766 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.474739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jhq" event={"ID":"03d46c54-4f83-41a9-9d99-e2f88aec3f4a","Type":"ContainerStarted","Data":"d96d965c71adb6bcec4326d8efbaabbe67b71ec1e231503bd4264b4bdb179e28"}
Jul 28 15:34:14.476052 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.476026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krkl4" event={"ID":"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2","Type":"ContainerStarted","Data":"218d91a17d7e1bcbc6d3b1754a1473fc8d530d921fbaa850c515e5e2aee5eab2"}
Jul 28 15:34:14.477360 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.477334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bnw82" event={"ID":"3fa94af5-82f1-42a5-9a55-3de4251112e1","Type":"ContainerStarted","Data":"22b0f629e00e0acde9f6e5e197f70c052d4e87a6ed4b52a8f62ea983ac799bc3"}
Jul 28 15:34:14.478609 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.478586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g" event={"ID":"616bd69c-c54b-4977-a225-afbbff19ec3c","Type":"ContainerStarted","Data":"3b5236feb7d91f93fa09f0d7bf56bfe6c3a41e1ffebab406854004fd5de0d8f7"}
Jul 28 15:34:14.483444 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:14.483424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:15.979514 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:15.979480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:15.980581 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:15.980559    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:15.987305 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:15.987024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:16.492661 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.492621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g" event={"ID":"616bd69c-c54b-4977-a225-afbbff19ec3c","Type":"ContainerStarted","Data":"7d7a98d9e787e5286f0a1a83e4ff4e4de5bd096d5a671d34e79ade3e5b392f9f"}
Jul 28 15:34:16.492847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.492690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g" event={"ID":"616bd69c-c54b-4977-a225-afbbff19ec3c","Type":"ContainerStarted","Data":"7d0e2fafea7adf1821bdb3bd4f4b9256b4b9da5b369dcd21dd78a79de5dfb3f4"}
Jul 28 15:34:16.497272 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.497026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krkl4" event={"ID":"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2","Type":"ContainerStarted","Data":"a0fe7828a15cb8979e3a73f854d990d0fae5127e7d92c0a8c1db5161c8785d5d"}
Jul 28 15:34:16.497272 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.497079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krkl4" event={"ID":"586a1d1d-6a0d-4cf6-9df0-41ee96f329a2","Type":"ContainerStarted","Data":"d99ab12f96b083f1dd9b6879362d0fe41e5b7dbf6a2c55f845895597c7680a61"}
Jul 28 15:34:16.501061 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.500902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:34:16.514014 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.513942    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-q9g5g" podStartSLOduration=2.435870874 podStartE2EDuration="4.513925569s" totalImagesPullingTime="2.078054695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:16.512888041 +0000 UTC m=+70.881540958" watchObservedRunningTime="2026-07-28 15:34:16.513925569 +0000 UTC m=+70.882578484"
Jul 28 15:34:16.537600 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.537493    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-krkl4" podStartSLOduration=69.265038692 podStartE2EDuration="1m10.53747604s" totalImagesPullingTime="1.272437348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:16.536796594 +0000 UTC m=+70.905449536" watchObservedRunningTime="2026-07-28 15:34:16.53747604 +0000 UTC m=+70.906128957"
Jul 28 15:34:16.588192 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.588152    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:16.633795 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.633764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:16.733802 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.733758    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:16.793769 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:16.793692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.261111 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:17.261067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.361346 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:17.361280    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.750834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:17.750804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.942804 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:17.942766    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:17.999164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:17.999118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"]
Jul 28 15:34:18.005742 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.005686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.008643 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.008610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jl6d5\""
Jul 28 15:34:18.008801 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.008713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 15:34:18.008801 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.008717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 15:34:18.014250 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.014226    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"]
Jul 28 15:34:18.021605 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.021577    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"]
Jul 28 15:34:18.024522 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.024502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.027263 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.027238    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 15:34:18.027409 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.027392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 15:34:18.027499 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.027441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mtw98\""
Jul 28 15:34:18.027557 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.027512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 15:34:18.036158 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.036100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"]
Jul 28 15:34:18.037174 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.037154    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rwt29"]
Jul 28 15:34:18.039400 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.039372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.041481 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.041412    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.042486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.041967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 15:34:18.042486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.042110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 15:34:18.042486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.042295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 15:34:18.042486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.042420    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t8s9r\""
Jul 28 15:34:18.069171 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/025f76d5-e003-4509-b00b-46bf82dfce35-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7ad692-43eb-497f-a82c-29f0f811e276-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.069596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxr9\" (UniqueName: \"kubernetes.io/projected/4d7ad692-43eb-497f-a82c-29f0f811e276-kube-api-access-xrxr9\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.069596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-root\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d678\" (UniqueName: \"kubernetes.io/projected/025f76d5-e003-4509-b00b-46bf82dfce35-kube-api-access-7d678\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-metrics-client-ca\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-wtmp\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpkf\" (UniqueName: \"kubernetes.io/projected/27598466-8844-4bff-af12-08a40ef6337f-kube-api-access-ldpkf\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-sys\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.069841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.069843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-textfile\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.136685 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.136622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.170457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.170457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7ad692-43eb-497f-a82c-29f0f811e276-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.170694 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxr9\" (UniqueName: \"kubernetes.io/projected/4d7ad692-43eb-497f-a82c-29f0f811e276-kube-api-access-xrxr9\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.170694 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.170694 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:18.170601    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 15:34:18.170694 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:18.170672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls podName:27598466-8844-4bff-af12-08a40ef6337f nodeName:}" failed. No retries permitted until 2026-07-28 15:34:18.670654145 +0000 UTC m=+73.039307039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls") pod "node-exporter-rwt29" (UID: "27598466-8844-4bff-af12-08a40ef6337f") : secret "node-exporter-tls" not found
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-root\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d678\" (UniqueName: \"kubernetes.io/projected/025f76d5-e003-4509-b00b-46bf82dfce35-kube-api-access-7d678\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.170999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-metrics-client-ca\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-wtmp\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpkf\" (UniqueName: \"kubernetes.io/projected/27598466-8844-4bff-af12-08a40ef6337f-kube-api-access-ldpkf\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-sys\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-textfile\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171287    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/025f76d5-e003-4509-b00b-46bf82dfce35-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.172753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7ad692-43eb-497f-a82c-29f0f811e276-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-wtmp\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.171732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-root\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.172383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025f76d5-e003-4509-b00b-46bf82dfce35-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.172488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/025f76d5-e003-4509-b00b-46bf82dfce35-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.172494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-metrics-client-ca\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.172559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27598466-8844-4bff-af12-08a40ef6337f-sys\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.172686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-textfile\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.173803 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.173284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.174627 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.174601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.174738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.174637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/025f76d5-e003-4509-b00b-46bf82dfce35-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.175047 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.175030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.176187 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.176168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7ad692-43eb-497f-a82c-29f0f811e276-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.176579 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.176561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.182640 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.182594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpkf\" (UniqueName: \"kubernetes.io/projected/27598466-8844-4bff-af12-08a40ef6337f-kube-api-access-ldpkf\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.183351 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.183327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxr9\" (UniqueName: \"kubernetes.io/projected/4d7ad692-43eb-497f-a82c-29f0f811e276-kube-api-access-xrxr9\") pod \"openshift-state-metrics-b8f49f5c6-pvzsl\" (UID: \"4d7ad692-43eb-497f-a82c-29f0f811e276\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.183450 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.183350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d678\" (UniqueName: \"kubernetes.io/projected/025f76d5-e003-4509-b00b-46bf82dfce35-kube-api-access-7d678\") pod \"kube-state-metrics-56d9f944b4-ks29s\" (UID: \"025f76d5-e003-4509-b00b-46bf82dfce35\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.240588 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.240519    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.318724 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.318638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"
Jul 28 15:34:18.335388 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.335360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"
Jul 28 15:34:18.337197 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.337171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.676911 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.676868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.679683 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.679653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27598466-8844-4bff-af12-08a40ef6337f-node-exporter-tls\") pod \"node-exporter-rwt29\" (UID: \"27598466-8844-4bff-af12-08a40ef6337f\") " pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:18.768564 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.768528    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.868798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.868764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:18.952744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:18.952650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwt29"
Jul 28 15:34:19.069710 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.069668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.158264 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.158222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.163888 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.163856    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:19.166962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.166935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.169832 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.169805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 15:34:19.170241 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170212    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 15:34:19.170424 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 15:34:19.170494 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170433    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 15:34:19.170691 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 15:34:19.170761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gfjwp\""
Jul 28 15:34:19.170761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 15:34:19.170822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.170716    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 15:34:19.171054 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.171037    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 15:34:19.172322 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.171605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 15:34:19.180766 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.180741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:19.277438 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.277355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:19.282436 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282599 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282599 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282599 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-config-out\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64nns\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-kube-api-access-64nns\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-web-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.282926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.282898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384335 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64nns\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-kube-api-access-64nns\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.384937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-web-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385035 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:19.384963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle podName:cd624046-1670-4924-98a5-58afbe7ade2a nodeName:}" failed. No retries permitted until 2026-07-28 15:34:19.884938853 +0000 UTC m=+74.253591747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "cd624046-1670-4924-98a5-58afbe7ade2a") : configmap references non-existent config key: ca-bundle.crt
Jul 28 15:34:19.385035 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.384998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385312 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385312 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-config-out\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.385312 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.385220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.386910 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.386847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.387847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.387800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.389032 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.389010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.389159 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.389049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.389212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.389165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.389256 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.389218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-web-config\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.390284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.390256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd624046-1670-4924-98a5-58afbe7ade2a-config-out\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.390457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.390437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.390632 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.390611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.391283 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.391241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd624046-1670-4924-98a5-58afbe7ade2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.394475 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.394452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64nns\" (UniqueName: \"kubernetes.io/projected/cd624046-1670-4924-98a5-58afbe7ade2a-kube-api-access-64nns\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.890608 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.890568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:19.891425 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:19.891402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd624046-1670-4924-98a5-58afbe7ade2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd624046-1670-4924-98a5-58afbe7ade2a\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:20.079058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.079028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 15:34:20.079529 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.079504    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"]
Jul 28 15:34:20.082628 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.082604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.087202 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.087179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 15:34:20.088316 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.088296    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 15:34:20.088419 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.088334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8j7ekr4d2h1d5\""
Jul 28 15:34:20.088419 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.088374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 15:34:20.088854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.088652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 15:34:20.088854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.088706    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w9v5t\""
Jul 28 15:34:20.089150 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.089112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 15:34:20.106784 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.106754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"]
Jul 28 15:34:20.192254 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-grpc-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192254 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrsxv\" (UniqueName: \"kubernetes.io/projected/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-kube-api-access-xrsxv\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-metrics-client-ca\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.192710 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.192508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.293678 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.293631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.293838 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.293691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.293838 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.293724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-metrics-client-ca\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.293838 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.293752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.294004 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.293980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.294063 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.294041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.294116 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.294095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-grpc-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.294903 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.294149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsxv\" (UniqueName: \"kubernetes.io/projected/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-kube-api-access-xrsxv\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.294903 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.294631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-metrics-client-ca\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.296954 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.296881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.297058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.297031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.297507 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.297483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.297625 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.297557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.297625 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.297611    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.297718 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.297683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-secret-grpc-tls\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.303654 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.303633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrsxv\" (UniqueName: \"kubernetes.io/projected/fbd3f6c6-6a21-4a33-aa90-cfe823272b5f-kube-api-access-xrsxv\") pod \"thanos-querier-6dfc4f7648-jfsnc\" (UID: \"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f\") " pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:20.394590 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:20.394550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:21.011065 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.011012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.020698 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:21.020650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27598466_8844_4bff_af12_08a40ef6337f.slice/crio-22c38ad965d55c6c936bddc209e91d36499673323c4c203489a82b02b814a0af WatchSource:0}: Error finding container 22c38ad965d55c6c936bddc209e91d36499673323c4c203489a82b02b814a0af: Status 404 returned error can't find the container with id 22c38ad965d55c6c936bddc209e91d36499673323c4c203489a82b02b814a0af
Jul 28 15:34:21.121293 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.118802    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.227723 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.227622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.240687 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.240655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl"]
Jul 28 15:34:21.245764 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:21.245666    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7ad692_43eb_497f_a82c_29f0f811e276.slice/crio-fded6f85bfbfa7185dd465a0bd283b1a51cbb16f7c0a837ae0f75d0aec42c628 WatchSource:0}: Error finding container fded6f85bfbfa7185dd465a0bd283b1a51cbb16f7c0a837ae0f75d0aec42c628: Status 404 returned error can't find the container with id fded6f85bfbfa7185dd465a0bd283b1a51cbb16f7c0a837ae0f75d0aec42c628
Jul 28 15:34:21.276699 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.276559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 15:34:21.284871 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:21.284270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd624046_1670_4924_98a5_58afbe7ade2a.slice/crio-9e2ea8391514400260894052a8fba54cf99ced0f6663747bd682a0e46395fc79 WatchSource:0}: Error finding container 9e2ea8391514400260894052a8fba54cf99ced0f6663747bd682a0e46395fc79: Status 404 returned error can't find the container with id 9e2ea8391514400260894052a8fba54cf99ced0f6663747bd682a0e46395fc79
Jul 28 15:34:21.460717 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.460682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.462189 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.462166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fcj9m"
Jul 28 15:34:21.485859 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.485729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s"]
Jul 28 15:34:21.489516 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:21.489484    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025f76d5_e003_4509_b00b_46bf82dfce35.slice/crio-3b7757a4c3cf271cd0a6ee3e936e5c512c827da51aed00c1a0cc72bd99796301 WatchSource:0}: Error finding container 3b7757a4c3cf271cd0a6ee3e936e5c512c827da51aed00c1a0cc72bd99796301: Status 404 returned error can't find the container with id 3b7757a4c3cf271cd0a6ee3e936e5c512c827da51aed00c1a0cc72bd99796301
Jul 28 15:34:21.490459 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.490436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"]
Jul 28 15:34:21.492699 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:21.492674    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd3f6c6_6a21_4a33_aa90_cfe823272b5f.slice/crio-397dc4002433213039f7509a09301da3ebd549cdc317bd98b3ccb8bc6df6f55c WatchSource:0}: Error finding container 397dc4002433213039f7509a09301da3ebd549cdc317bd98b3ccb8bc6df6f55c: Status 404 returned error can't find the container with id 397dc4002433213039f7509a09301da3ebd549cdc317bd98b3ccb8bc6df6f55c
Jul 28 15:34:21.516706 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.516670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s" event={"ID":"025f76d5-e003-4509-b00b-46bf82dfce35","Type":"ContainerStarted","Data":"3b7757a4c3cf271cd0a6ee3e936e5c512c827da51aed00c1a0cc72bd99796301"}
Jul 28 15:34:21.518854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.518820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jstzl" event={"ID":"3734d978-9bc8-4025-99ee-f847754332e5","Type":"ContainerStarted","Data":"f0001833cfca8e1540e18beffd7ba15f0d41b3d82ba4c982c563683e55c5020b"}
Jul 28 15:34:21.520284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.520250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"9e2ea8391514400260894052a8fba54cf99ced0f6663747bd682a0e46395fc79"}
Jul 28 15:34:21.522247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.522224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl" event={"ID":"4d7ad692-43eb-497f-a82c-29f0f811e276","Type":"ContainerStarted","Data":"fd9005819abef1d3a5742cc241fa857b8911f3f81c957a02e031906639ac8b45"}
Jul 28 15:34:21.522350 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.522275    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl" event={"ID":"4d7ad692-43eb-497f-a82c-29f0f811e276","Type":"ContainerStarted","Data":"b122f7e73f3864499b53943af898b2e54f348262083e5025e71e0e5b97bf91d9"}
Jul 28 15:34:21.522350 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.522304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl" event={"ID":"4d7ad692-43eb-497f-a82c-29f0f811e276","Type":"ContainerStarted","Data":"fded6f85bfbfa7185dd465a0bd283b1a51cbb16f7c0a837ae0f75d0aec42c628"}
Jul 28 15:34:21.524734 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.524709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6jhq" event={"ID":"03d46c54-4f83-41a9-9d99-e2f88aec3f4a","Type":"ContainerStarted","Data":"49411f220cf7c7353d0ccc4a17e2788f53cc6d60cb1c11e0948c4bbc9e38d524"}
Jul 28 15:34:21.527262 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.527213    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bnw82" event={"ID":"3fa94af5-82f1-42a5-9a55-3de4251112e1","Type":"ContainerStarted","Data":"2c6789ffec6c48a86e9925a1b5a69ee12abd35a2eeff06e2acfad75d3ba92096"}
Jul 28 15:34:21.527435 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.527336    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:21.528563 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.528506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwt29" event={"ID":"27598466-8844-4bff-af12-08a40ef6337f","Type":"ContainerStarted","Data":"22c38ad965d55c6c936bddc209e91d36499673323c4c203489a82b02b814a0af"}
Jul 28 15:34:21.529759 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.529739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"397dc4002433213039f7509a09301da3ebd549cdc317bd98b3ccb8bc6df6f55c"}
Jul 28 15:34:21.540165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.540064    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jstzl" podStartSLOduration=68.719981922 podStartE2EDuration="1m15.540047464s" totalImagesPullingTime="6.820065542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:21.538968927 +0000 UTC m=+75.907621850" watchObservedRunningTime="2026-07-28 15:34:21.540047464 +0000 UTC m=+75.908700382"
Jul 28 15:34:21.556455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.556427    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:21.595177 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.595090    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h6jhq" podStartSLOduration=1.330619196 podStartE2EDuration="19.595067823s" totalImagesPullingTime="18.264448627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:21.591684684 +0000 UTC m=+75.960337599" watchObservedRunningTime="2026-07-28 15:34:21.595067823 +0000 UTC m=+75.963720740"
Jul 28 15:34:21.595669 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.595638    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bnw82" podStartSLOduration=68.540136903 podStartE2EDuration="1m15.595628774s" totalImagesPullingTime="7.055491871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:21.567960916 +0000 UTC m=+75.936613834" watchObservedRunningTime="2026-07-28 15:34:21.595628774 +0000 UTC m=+75.964281725"
Jul 28 15:34:21.652862 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:21.652831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.147428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.147376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.262787 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.262674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.343473 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.342625    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b97796b5-xrvvh"]
Jul 28 15:34:22.345218 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.345183    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.347945 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.347366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.350428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.350222    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 15:34:22.352864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.351603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xtdhv\""
Jul 28 15:34:22.352864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.352015    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5tgr36u6eol8u\""
Jul 28 15:34:22.352864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.352304    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 15:34:22.352864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.352328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 15:34:22.352864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.352505    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 15:34:22.359461 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.359434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b97796b5-xrvvh"]
Jul 28 15:34:22.424207 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424367 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424230    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-tls\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424367 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-client-certs\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424367 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6dc\" (UniqueName: \"kubernetes.io/projected/549cd043-a164-4b0a-be24-a4260992053f-kube-api-access-pw6dc\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424526 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-metrics-server-audit-profiles\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424526 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/549cd043-a164-4b0a-be24-a4260992053f-audit-log\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.424526 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.424465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-client-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.524943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/549cd043-a164-4b0a-be24-a4260992053f-audit-log\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.524993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-client-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-tls\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-client-certs\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6dc\" (UniqueName: \"kubernetes.io/projected/549cd043-a164-4b0a-be24-a4260992053f-kube-api-access-pw6dc\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.525447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-metrics-server-audit-profiles\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.526220 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/549cd043-a164-4b0a-be24-a4260992053f-audit-log\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.526220 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.525987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.527323 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.527276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/549cd043-a164-4b0a-be24-a4260992053f-metrics-server-audit-profiles\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.529927 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.529120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-client-ca-bundle\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.536678 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.530983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-tls\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.536678 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.534040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/549cd043-a164-4b0a-be24-a4260992053f-secret-metrics-server-client-certs\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.541987 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.541656    2580 generic.go:356] "Generic (PLEG): container finished" podID="27598466-8844-4bff-af12-08a40ef6337f" containerID="b1e3d321a7973d666bce7bd9aa6387551958f393dcc1d6a621c86f14b95d14ed" exitCode=0
Jul 28 15:34:22.541987 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.541761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwt29" event={"ID":"27598466-8844-4bff-af12-08a40ef6337f","Type":"ContainerDied","Data":"b1e3d321a7973d666bce7bd9aa6387551958f393dcc1d6a621c86f14b95d14ed"}
Jul 28 15:34:22.547074 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.547036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6dc\" (UniqueName: \"kubernetes.io/projected/549cd043-a164-4b0a-be24-a4260992053f-kube-api-access-pw6dc\") pod \"metrics-server-9b97796b5-xrvvh\" (UID: \"549cd043-a164-4b0a-be24-a4260992053f\") " pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.667059 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.667022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:22.785495 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.785426    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.792609 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.792576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"]
Jul 28 15:34:22.796624 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.796052    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:22.798793 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.798768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 15:34:22.798926 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.798884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7g2r5\""
Jul 28 15:34:22.807249 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.807189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"]
Jul 28 15:34:22.875384 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.875343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:22.928470 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.928435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:22.928655 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:22.928516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/2a8c1491-a611-40f0-a274-a345088606ee-kube-api-access-j4w7f\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.029695 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.029660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.029873 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.029735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/2a8c1491-a611-40f0-a274-a345088606ee-kube-api-access-j4w7f\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.029873 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:23.029833    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 15:34:23.029976 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:23.029905    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert podName:2a8c1491-a611-40f0-a274-a345088606ee nodeName:}" failed. No retries permitted until 2026-07-28 15:34:23.529885494 +0000 UTC m=+77.898538390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-sdvdz" (UID: "2a8c1491-a611-40f0-a274-a345088606ee") : secret "monitoring-plugin-cert" not found
Jul 28 15:34:23.046195 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.046098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4w7f\" (UniqueName: \"kubernetes.io/projected/2a8c1491-a611-40f0-a274-a345088606ee-kube-api-access-j4w7f\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.307637 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.307492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:23.534716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.534676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.537491 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.537453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a8c1491-a611-40f0-a274-a345088606ee-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-sdvdz\" (UID: \"2a8c1491-a611-40f0-a274-a345088606ee\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:23.710249 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:23.710211    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:24.145991 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.145965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b97796b5-xrvvh"]
Jul 28 15:34:24.152609 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:24.152581    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549cd043_a164_4b0a_be24_a4260992053f.slice/crio-14e9dc39ea5ffccf50c284b2bced5381485ab9c00dd3a9d7ee0b10283e979560 WatchSource:0}: Error finding container 14e9dc39ea5ffccf50c284b2bced5381485ab9c00dd3a9d7ee0b10283e979560: Status 404 returned error can't find the container with id 14e9dc39ea5ffccf50c284b2bced5381485ab9c00dd3a9d7ee0b10283e979560
Jul 28 15:34:24.159588 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.158924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"]
Jul 28 15:34:24.166532 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:24.166494    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8c1491_a611_40f0_a274_a345088606ee.slice/crio-ce853cf9905e48c170bafe2c4cd5ee5a210dcb28c60fd65c9bc7a1643ea9f936 WatchSource:0}: Error finding container ce853cf9905e48c170bafe2c4cd5ee5a210dcb28c60fd65c9bc7a1643ea9f936: Status 404 returned error can't find the container with id ce853cf9905e48c170bafe2c4cd5ee5a210dcb28c60fd65c9bc7a1643ea9f936
Jul 28 15:34:24.358776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.358526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.359538 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.359389    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 15:34:24.363575 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.363552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.368955    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.369255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.369450    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.370361    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2ht4o65l5hvju\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.370568    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.370784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.371113    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.371392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372030    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372082    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 15:34:24.372244 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372213    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xb8mg\""
Jul 28 15:34:24.372928 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372442    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 15:34:24.372928 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 15:34:24.372928 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.372827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 15:34:24.386535 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.386513    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 15:34:24.391226 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.390504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 15:34:24.443583 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.443765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.443765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.443765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.443765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.443765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.443998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.444017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.444047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.444061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pnf\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-kube-api-access-46pnf\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.444109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.444479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.444171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.545528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.545528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pnf\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-kube-api-access-46pnf\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.545528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.545670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546165 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546927 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546927 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.546927 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.546552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.548947 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.548860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.549474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.549810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.549857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.549913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.550074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.550105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550458 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.550309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-config\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.550869 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.550535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.551320 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.551240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.551320 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.551305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.551918 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.551897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.551962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.551937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d2838cb-f368-421a-b994-ec4d0172ddd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.552049 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.552028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d2838cb-f368-421a-b994-ec4d0172ddd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.552751 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.552708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.553567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.553520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.555579 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.555540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d2838cb-f368-421a-b994-ec4d0172ddd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.557894 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.557869    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd624046-1670-4924-98a5-58afbe7ade2a" containerID="7db31ec29b45003683a9b7a07f48c48d3be2758646695ee8e2c874198007105e" exitCode=0
Jul 28 15:34:24.558063 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.558022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerDied","Data":"7db31ec29b45003683a9b7a07f48c48d3be2758646695ee8e2c874198007105e"}
Jul 28 15:34:24.558616 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.558591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pnf\" (UniqueName: \"kubernetes.io/projected/1d2838cb-f368-421a-b994-ec4d0172ddd0-kube-api-access-46pnf\") pod \"prometheus-k8s-0\" (UID: \"1d2838cb-f368-421a-b994-ec4d0172ddd0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.560714 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.560688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl" event={"ID":"4d7ad692-43eb-497f-a82c-29f0f811e276","Type":"ContainerStarted","Data":"976a1420c2802b510a28a20e45d225fa75c263799017efda33ec9ad3ab0043ad"}
Jul 28 15:34:24.565756 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.565671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwt29" event={"ID":"27598466-8844-4bff-af12-08a40ef6337f","Type":"ContainerStarted","Data":"fd7b23fccd9fe162e91a6c9981eb4ab2f0bdf78c60ea11f96616584993fa85ae"}
Jul 28 15:34:24.565756 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.565723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwt29" event={"ID":"27598466-8844-4bff-af12-08a40ef6337f","Type":"ContainerStarted","Data":"15e8d272a1578da75c20b2bb0aecda985546fb3f9531f25dd9bd86916cee0814"}
Jul 28 15:34:24.567078 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.567057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz" event={"ID":"2a8c1491-a611-40f0-a274-a345088606ee","Type":"ContainerStarted","Data":"ce853cf9905e48c170bafe2c4cd5ee5a210dcb28c60fd65c9bc7a1643ea9f936"}
Jul 28 15:34:24.572939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.572773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"a5e0a1a91eb8d71667833ee595a0b99c814e9661afb226495a6dd34f989153ac"}
Jul 28 15:34:24.572939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.572843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"8b9cb0395ac75e0dd4808efadc04c30e7d981e8a838d1a8e0a56b956db7a1dc0"}
Jul 28 15:34:24.572939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.572882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"fa9796f7f7e6a41f52acf4f8c01761d2df6cbfa5e85d2f1ecb587ae15d1d37cc"}
Jul 28 15:34:24.576519 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.576489    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s" event={"ID":"025f76d5-e003-4509-b00b-46bf82dfce35","Type":"ContainerStarted","Data":"652b87770e5d61ae4044498c2deb398fb72cf3a1b7c6d256fa54be923e047b8b"}
Jul 28 15:34:24.576618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.576550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s" event={"ID":"025f76d5-e003-4509-b00b-46bf82dfce35","Type":"ContainerStarted","Data":"b12d1a6f6edbe8e46bc565d40602ec81b3a22ed4ad4abd45d08510fa091885a1"}
Jul 28 15:34:24.576618 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.576578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s" event={"ID":"025f76d5-e003-4509-b00b-46bf82dfce35","Type":"ContainerStarted","Data":"efd49b6b1a689634b0c17ea6258766ce616169aaf546dd3ba6f65fb886b1ffd5"}
Jul 28 15:34:24.583355 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.583326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh" event={"ID":"549cd043-a164-4b0a-be24-a4260992053f","Type":"ContainerStarted","Data":"14e9dc39ea5ffccf50c284b2bced5381485ab9c00dd3a9d7ee0b10283e979560"}
Jul 28 15:34:24.589846 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.589816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.605183 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.605110    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-pvzsl" podStartSLOduration=5.041020483 podStartE2EDuration="7.605092819s" totalImagesPullingTime="2.564072336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:24.603475809 +0000 UTC m=+78.972128727" watchObservedRunningTime="2026-07-28 15:34:24.605092819 +0000 UTC m=+78.973745735"
Jul 28 15:34:24.625814 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.625742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rwt29" podStartSLOduration=5.619897498 podStartE2EDuration="6.625719035s" totalImagesPullingTime="934.350633ms" totalInitContainerRuntime="71.470904ms" isStatefulPod=true podCreationTimestamp="2026-07-28 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:24.623896614 +0000 UTC m=+78.992549537" watchObservedRunningTime="2026-07-28 15:34:24.625719035 +0000 UTC m=+78.994371945"
Jul 28 15:34:24.643959 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.643895    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ks29s" podStartSLOduration=5.17742106 podStartE2EDuration="7.643874424s" totalImagesPullingTime="2.466453364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:24.642220682 +0000 UTC m=+79.010873600" watchObservedRunningTime="2026-07-28 15:34:24.643874424 +0000 UTC m=+79.012527341"
Jul 28 15:34:24.683682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.683641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:24.814163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.814064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.850996 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.848883    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 15:34:24.853170 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:34:24.852851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2838cb_f368_421a_b994_ec4d0172ddd0.slice/crio-a828abe7f3eeacb42c52ec103dd8f0a244c55834c6b6e5e67fa41850f3780231 WatchSource:0}: Error finding container a828abe7f3eeacb42c52ec103dd8f0a244c55834c6b6e5e67fa41850f3780231: Status 404 returned error can't find the container with id a828abe7f3eeacb42c52ec103dd8f0a244c55834c6b6e5e67fa41850f3780231
Jul 28 15:34:24.886324 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.886241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:24.990139 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:24.990088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:25.592163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:25.591286    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d2838cb-f368-421a-b994-ec4d0172ddd0" containerID="05f5543deff4bb00de0ee7d6ab4394521e2d1f986d4bcc45a70a13c598af84ed" exitCode=0
Jul 28 15:34:25.592163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:25.591752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerDied","Data":"05f5543deff4bb00de0ee7d6ab4394521e2d1f986d4bcc45a70a13c598af84ed"}
Jul 28 15:34:25.592163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:25.591881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"a828abe7f3eeacb42c52ec103dd8f0a244c55834c6b6e5e67fa41850f3780231"}
Jul 28 15:34:27.261240 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.261013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.332087 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.332053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:27.612211 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"79f5b64f22ccc1741f5b424612e1af16f6aab3271e9f2c265b2163fb3d7f5a63"}
Jul 28 15:34:27.612365 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"5ced538ac6890fa9fbd57310a6036c8da797eabf9c0ed1971660b4ac32f11050"}
Jul 28 15:34:27.612365 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"1c1eeecc5e4ac7a228803346f8ae3a31fb2e9e6cd49e7feee99f361486b16294"}
Jul 28 15:34:27.612365 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"be573561ef1511ce6f3cfebf14fcd5b407691f651ef52e3471b036da5e153927"}
Jul 28 15:34:27.612564 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"0e16b6c481b684ba6b413427d86668aab1bc1cc5a93223c00c78a9132583ba0a"}
Jul 28 15:34:27.612564 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.612425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd624046-1670-4924-98a5-58afbe7ade2a","Type":"ContainerStarted","Data":"89439cd1d11a279fb6da72982f023fd863b115bf3e0968997d44fbf6215b5a86"}
Jul 28 15:34:27.614211 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.614170    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz" event={"ID":"2a8c1491-a611-40f0-a274-a345088606ee","Type":"ContainerStarted","Data":"850112c3a2acf428cfa52adb6fb0071c30a16890531833d1fb01c40aaab76958"}
Jul 28 15:34:27.614426 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.614361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:27.615913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.615885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh" event={"ID":"549cd043-a164-4b0a-be24-a4260992053f","Type":"ContainerStarted","Data":"37366037459e5451ff0dfb07c1f0b232f9c9375cca0cc5e251219ded982c9114"}
Jul 28 15:34:27.620269 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.620246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz"
Jul 28 15:34:27.620798 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.620771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"cad87c9d800d23c09f375e9d2bc0cee401e2111e8ffec5dd54343b533b5631ac"}
Jul 28 15:34:27.620877 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.620841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"4f440d5518b92a2118d943e58b34c8a77f1082d724936f9ca34f55dd007f0f96"}
Jul 28 15:34:27.620931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.620888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" event={"ID":"fbd3f6c6-6a21-4a33-aa90-cfe823272b5f","Type":"ContainerStarted","Data":"19aa491c2c9717ea49a95bc97b55d7c2a39f851c0e5dcc5641b543447bf2d04f"}
Jul 28 15:34:27.620989 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.620951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:27.642892 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.642838    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.791657773 podStartE2EDuration="8.642822302s" totalImagesPullingTime="4.832818889s" totalInitContainerRuntime="18.34564ms" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:27.640588345 +0000 UTC m=+82.009241288" watchObservedRunningTime="2026-07-28 15:34:27.642822302 +0000 UTC m=+82.011475219"
Jul 28 15:34:27.661003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.660950    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh" podStartSLOduration=3.153958929 podStartE2EDuration="5.660932106s" totalImagesPullingTime="2.506973177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:27.660044922 +0000 UTC m=+82.028697841" watchObservedRunningTime="2026-07-28 15:34:27.660932106 +0000 UTC m=+82.029585015"
Jul 28 15:34:27.689434 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:27.688155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc" podStartSLOduration=2.912955004 podStartE2EDuration="7.688106611s" totalImagesPullingTime="4.775151607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:27.686790447 +0000 UTC m=+82.055443376" watchObservedRunningTime="2026-07-28 15:34:27.688106611 +0000 UTC m=+82.056759529"
Jul 28 15:34:28.249588 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:28.249549    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:28.318646 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:28.318613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:29.632336 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:29.632304    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"bfb42dfdd229fcdb00330d9166d16ac87a04cec6871d3785c0644c1121079493"}
Jul 28 15:34:30.634428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.634388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:30.640058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.640030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"219857bb4efaa07680525756e87873ca9d25c5e08a7d0af3d64c2bde0156b627"}
Jul 28 15:34:30.640238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.640093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"4abc0d02dea387363c2a537a51702f356a3874544c6a3e7009f4b4459b5e118f"}
Jul 28 15:34:30.640238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.640121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"77f4e991336a98065a021fc4c5a34850d20405cd8bfba641cd174cbf72e7a380"}
Jul 28 15:34:30.640238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.640164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"96a11eacfd75d7f8598c35de5eed526c034e988af46ab7424a766b02ac0cf9a9"}
Jul 28 15:34:30.640238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.640187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d2838cb-f368-421a-b994-ec4d0172ddd0","Type":"ContainerStarted","Data":"44785696e44b8744693f39502ab4bf1d12cadc9277276a0891c74bad231e6eda"}
Jul 28 15:34:30.676543 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.676477    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-sdvdz" podStartSLOduration=6.187605257 podStartE2EDuration="8.676455388s" totalImagesPullingTime="2.488850131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:27.706923352 +0000 UTC m=+82.075576269" watchObservedRunningTime="2026-07-28 15:34:30.676455388 +0000 UTC m=+85.045108305"
Jul 28 15:34:30.676917 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.676863    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.717161667 podStartE2EDuration="6.676846198s" totalImagesPullingTime="3.936305366s" totalInitContainerRuntime="23.379165ms" isStatefulPod=false podCreationTimestamp="2026-07-28 15:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:34:30.674504379 +0000 UTC m=+85.043157296" watchObservedRunningTime="2026-07-28 15:34:30.676846198 +0000 UTC m=+85.045499115"
Jul 28 15:34:30.704678 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:30.704647    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:31.268566 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:31.268535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:31.335367 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:31.335339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:33.160044 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:33.160009    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:33.250164 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:33.250112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:33.631856 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:33.631831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6dfc4f7648-jfsnc"
Jul 28 15:34:34.683785 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:34.683753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:34:38.857431 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:38.857380    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:34:38.962743 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:38.962713    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:39.063152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:39.063101    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:39.120755 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:39.120684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:41.622345 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:41.622269    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c9bd44d8b-vs7xc" podUID="ec385968-1fcf-44df-a9f3-91d2eb414e00" containerName="console" containerID="cri-o://73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16" gracePeriod=15
Jul 28 15:34:42.434963 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.434929    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:42.525326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525500 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525500 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ckql\" (UniqueName: \"kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525500 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525439    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525500 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525712 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config\") pod \"ec385968-1fcf-44df-a9f3-91d2eb414e00\" (UID: \"ec385968-1fcf-44df-a9f3-91d2eb414e00\") "
Jul 28 15:34:42.525861 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:34:42.525981 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config" (OuterVolumeSpecName: "console-config") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:34:42.525981 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.525950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:34:42.527991 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.527931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:34:42.527991 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.527960    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql" (OuterVolumeSpecName: "kube-api-access-2ckql") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "kube-api-access-2ckql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:34:42.527991 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.527977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec385968-1fcf-44df-a9f3-91d2eb414e00" (UID: "ec385968-1fcf-44df-a9f3-91d2eb414e00"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:34:42.626587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626557    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-oauth-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.626587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626584    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-oauth-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.626587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626593    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.627000 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626601    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec385968-1fcf-44df-a9f3-91d2eb414e00-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.627000 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626611    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec385968-1fcf-44df-a9f3-91d2eb414e00-console-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.627000 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.626621    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckql\" (UniqueName: \"kubernetes.io/projected/ec385968-1fcf-44df-a9f3-91d2eb414e00-kube-api-access-2ckql\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:34:42.667492 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.667457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:42.667634 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.667527    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:34:42.691830 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.691790    2580 generic.go:356] "Generic (PLEG): container finished" podID="ec385968-1fcf-44df-a9f3-91d2eb414e00" containerID="73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16" exitCode=0
Jul 28 15:34:42.691978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.691836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd44d8b-vs7xc"
Jul 28 15:34:42.691978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.691867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd44d8b-vs7xc" event={"ID":"ec385968-1fcf-44df-a9f3-91d2eb414e00","Type":"ContainerDied","Data":"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"}
Jul 28 15:34:42.691978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.691918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd44d8b-vs7xc" event={"ID":"ec385968-1fcf-44df-a9f3-91d2eb414e00","Type":"ContainerDied","Data":"3432c52ef6149143088d0e881699cd98ed273b6ef144973fec8fa094d0e22f63"}
Jul 28 15:34:42.691978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.691945    2580 scope.go:117] "RemoveContainer" containerID="73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"
Jul 28 15:34:42.701770 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.701753    2580 scope.go:117] "RemoveContainer" containerID="73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"
Jul 28 15:34:42.702055 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:34:42.702033    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16\": container with ID starting with 73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16 not found: ID does not exist" containerID="73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"
Jul 28 15:34:42.702167 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.702062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16"} err="failed to get container status \"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16\": rpc error: code = NotFound desc = could not find container \"73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16\": container with ID starting with 73883fac9b84703cd40d31f5ac2b473e31e5254cdca61b77cb4853fc66296c16 not found: ID does not exist"
Jul 28 15:34:42.712793 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.712769    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:42.716761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:42.716740    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9bd44d8b-vs7xc"]
Jul 28 15:34:43.250912 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:43.250878    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:43.318780 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:43.318725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:44.186076 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:44.186035    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec385968-1fcf-44df-a9f3-91d2eb414e00" path="/var/lib/kubelet/pods/ec385968-1fcf-44df-a9f3-91d2eb414e00/volumes"
Jul 28 15:34:52.545819 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:52.545784    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bnw82"
Jul 28 15:34:54.647488 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:54.647461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:34:54.725092 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:34:54.725030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:02.672533 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:02.672506    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:35:02.676271 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:02.676196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b97796b5-xrvvh"
Jul 28 15:35:03.877443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:03.877401    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-769b9fbd8c-4nxqw" podUID="4c9f8714-2aae-4a52-b6ef-04654f186af3" containerName="console" containerID="cri-o://0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469" gracePeriod=15
Jul 28 15:35:04.125809 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.125781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:35:04.208771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlmgh\" (UniqueName: \"kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.208771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.208771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.209032 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.209032 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.209032 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.209032 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.208860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config\") pod \"4c9f8714-2aae-4a52-b6ef-04654f186af3\" (UID: \"4c9f8714-2aae-4a52-b6ef-04654f186af3\") "
Jul 28 15:35:04.209301 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.209276    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:04.209361 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.209336    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:04.209361 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.209343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:04.209434 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.209407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config" (OuterVolumeSpecName: "console-config") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:35:04.211167 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.211117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:04.211167 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.211154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh" (OuterVolumeSpecName: "kube-api-access-hlmgh") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "kube-api-access-hlmgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:35:04.211316 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.211221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c9f8714-2aae-4a52-b6ef-04654f186af3" (UID: "4c9f8714-2aae-4a52-b6ef-04654f186af3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:35:04.309705 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309653    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmgh\" (UniqueName: \"kubernetes.io/projected/4c9f8714-2aae-4a52-b6ef-04654f186af3-kube-api-access-hlmgh\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309705 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309702    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-trusted-ca-bundle\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309705 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309712    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309705 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309721    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-oauth-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309970 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309730    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309970 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309738    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-oauth-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.309970 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.309748    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c9f8714-2aae-4a52-b6ef-04654f186af3-console-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:35:04.777621 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.777588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:04.781727 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.781695    2580 generic.go:356] "Generic (PLEG): container finished" podID="4c9f8714-2aae-4a52-b6ef-04654f186af3" containerID="0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469" exitCode=0
Jul 28 15:35:04.781871 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.781746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-769b9fbd8c-4nxqw"
Jul 28 15:35:04.781871 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.781777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769b9fbd8c-4nxqw" event={"ID":"4c9f8714-2aae-4a52-b6ef-04654f186af3","Type":"ContainerDied","Data":"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"}
Jul 28 15:35:04.781871 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.781831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-769b9fbd8c-4nxqw" event={"ID":"4c9f8714-2aae-4a52-b6ef-04654f186af3","Type":"ContainerDied","Data":"9f9d09620a0f32d208590fb44b46a6f7ede8a309428fda26f3a1551b1bca28b8"}
Jul 28 15:35:04.781871 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.781853    2580 scope.go:117] "RemoveContainer" containerID="0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"
Jul 28 15:35:04.790571 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.790555    2580 scope.go:117] "RemoveContainer" containerID="0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"
Jul 28 15:35:04.790828 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:35:04.790810    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469\": container with ID starting with 0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469 not found: ID does not exist" containerID="0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"
Jul 28 15:35:04.790905 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.790841    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469"} err="failed to get container status \"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469\": rpc error: code = NotFound desc = could not find container \"0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469\": container with ID starting with 0940371e1f8d540fb3e1614bf50d71beb12b8bc63e083ca0a0b9b7d2fec68469 not found: ID does not exist"
Jul 28 15:35:04.803757 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.803732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:35:04.808357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:04.808332    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-769b9fbd8c-4nxqw"]
Jul 28 15:35:06.186147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:06.186091    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9f8714-2aae-4a52-b6ef-04654f186af3" path="/var/lib/kubelet/pods/4c9f8714-2aae-4a52-b6ef-04654f186af3/volumes"
Jul 28 15:35:07.585470 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:07.585435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:07.650267 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:07.650237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:12.096576 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:12.096525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:24.684300 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:24.684254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:35:24.705301 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:24.705260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:35:24.885400 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:24.885361    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 15:35:37.302393 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:37.302352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:37.714642 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:37.714582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:42.373229 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.373195    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-749f86dc87-5mwwf"]
Jul 28 15:35:42.374187 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374158    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec385968-1fcf-44df-a9f3-91d2eb414e00" containerName="console"
Jul 28 15:35:42.374187 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374185    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec385968-1fcf-44df-a9f3-91d2eb414e00" containerName="console"
Jul 28 15:35:42.374357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374220    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9f8714-2aae-4a52-b6ef-04654f186af3" containerName="console"
Jul 28 15:35:42.374357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374229    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9f8714-2aae-4a52-b6ef-04654f186af3" containerName="console"
Jul 28 15:35:42.374357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374318    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9f8714-2aae-4a52-b6ef-04654f186af3" containerName="console"
Jul 28 15:35:42.374357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.374328    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec385968-1fcf-44df-a9f3-91d2eb414e00" containerName="console"
Jul 28 15:35:42.378465 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.378438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.381387 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381353    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 15:35:42.381387 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 15:35:42.381567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 15:35:42.381567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381418    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jlgtd\""
Jul 28 15:35:42.381567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 15:35:42.381729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.381663    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 15:35:42.387727 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.386921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 15:35:42.389824 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.389806    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-749f86dc87-5mwwf"]
Jul 28 15:35:42.448380 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-metrics-client-ca\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448380 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-serving-certs-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448647 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scv9g\" (UniqueName: \"kubernetes.io/projected/b8671285-9c62-4350-a9e1-e3e728409d0f-kube-api-access-scv9g\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448647 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448647 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448647 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-federate-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448827 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.448827 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.448718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549642 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549642 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-metrics-client-ca\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-serving-certs-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scv9g\" (UniqueName: \"kubernetes.io/projected/b8671285-9c62-4350-a9e1-e3e728409d0f-kube-api-access-scv9g\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549954 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.549954 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.550053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.549959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-federate-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.550053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.550001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.550444 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.550415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-metrics-client-ca\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.550444 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.550427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-serving-certs-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.551088 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.551067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.552528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.552499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.552528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.552503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.552936 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.552919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-federate-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.552997 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.552950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b8671285-9c62-4350-a9e1-e3e728409d0f-telemeter-client-tls\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.558563 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.558540    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scv9g\" (UniqueName: \"kubernetes.io/projected/b8671285-9c62-4350-a9e1-e3e728409d0f-kube-api-access-scv9g\") pod \"telemeter-client-749f86dc87-5mwwf\" (UID: \"b8671285-9c62-4350-a9e1-e3e728409d0f\") " pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.690603 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.690527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf"
Jul 28 15:35:42.856650 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.856618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-749f86dc87-5mwwf"]
Jul 28 15:35:42.859957 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:35:42.859932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8671285_9c62_4350_a9e1_e3e728409d0f.slice/crio-96ed59243beb866addb33ef5fc416b1c306b2a1fbf804562dfd111c1cf7d9cba WatchSource:0}: Error finding container 96ed59243beb866addb33ef5fc416b1c306b2a1fbf804562dfd111c1cf7d9cba: Status 404 returned error can't find the container with id 96ed59243beb866addb33ef5fc416b1c306b2a1fbf804562dfd111c1cf7d9cba
Jul 28 15:35:42.939864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:42.939826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf" event={"ID":"b8671285-9c62-4350-a9e1-e3e728409d0f","Type":"ContainerStarted","Data":"96ed59243beb866addb33ef5fc416b1c306b2a1fbf804562dfd111c1cf7d9cba"}
Jul 28 15:35:44.950792 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:44.950754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf" event={"ID":"b8671285-9c62-4350-a9e1-e3e728409d0f","Type":"ContainerStarted","Data":"dc6ea858349973aacff558840fdd77ec0b7f2cc24c8a764ebed26f439cee2fb4"}
Jul 28 15:35:44.951247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:44.950812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf" event={"ID":"b8671285-9c62-4350-a9e1-e3e728409d0f","Type":"ContainerStarted","Data":"61d7921333ae17607fae16e7aa445d289bde86b670d51941d66a5b025267061a"}
Jul 28 15:35:44.951247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:44.950832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf" event={"ID":"b8671285-9c62-4350-a9e1-e3e728409d0f","Type":"ContainerStarted","Data":"26b3d3ea1f0447f1b862d808a445186ade1942c4f3de6c71bb18bf2c898f6667"}
Jul 28 15:35:44.975514 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:44.975464    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-749f86dc87-5mwwf" podStartSLOduration=1.274625721 podStartE2EDuration="2.975447601s" totalImagesPullingTime="1.70082188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:35:44.973678848 +0000 UTC m=+159.342331766" watchObservedRunningTime="2026-07-28 15:35:44.975447601 +0000 UTC m=+159.344100516"
Jul 28 15:35:45.903422 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:45.903388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:45.969254 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:45.969224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:46.108047 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:46.108012    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:46.211563 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:46.211439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:46.646841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:46.646803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:46.707283 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:46.707250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:52.537543 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:52.537504    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:35:52.599313 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:35:52.599274    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:03.467147 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:03.467053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:04.244076 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:04.244039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:04.304757 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:04.304725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:04.368575 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:04.368542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:04.484097 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:04.484059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:04.584744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:04.584707    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:13.255810 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:13.255772    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:13.356867 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:13.356832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:15.295288 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:15.295254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:15.393098 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:15.393061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:42.175515 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:42.175477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:36:42.240317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:36:42.240284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.078060 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:37.077979    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.175362 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:37.175327    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.442440 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:37.442403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.542399 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:37.542365    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:37.645416 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:37.645389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:37:43.961173 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:37:43.961080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:04.831271 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:04.831226    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:06.035873 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:06.035846    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 15:38:06.405307 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:06.405271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:10.470996 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:10.470957    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:10.776276 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:10.776088    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.046193 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.046083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.109771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.109736    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.351635 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.351604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.450471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.450436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.541339 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.541307    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.641030 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.640935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.833616 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.833577    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:11.964634 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:11.964546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:12.167390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:12.167361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:12.264479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:12.264390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:12.802469 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:12.802440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:13.024443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:13.024410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:50.440371 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:50.440333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:38:50.537531 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:38:50.537492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:06.431002 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:06.430886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:28.944783 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:28.944737    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.016755 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.016720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.105570 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.105537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.373363 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.373331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.468248 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.468219    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.590593 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.590561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.691326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.691222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:29.791827 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:29.791799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:35.144800 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:35.144759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:39:35.219880 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:39:35.219843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:45.769861 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.769811    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:45.805053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.805016    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-krzcb"]
Jul 28 15:40:45.807287 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.807267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.809985 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.809963    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 15:40:45.810100 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.810054    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 15:40:45.810214 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.810202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-xf47m\""
Jul 28 15:40:45.811180 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.811158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 15:40:45.817487 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.817459    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-krzcb"]
Jul 28 15:40:45.847712 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.847667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:45.876218 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.876191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxctk\" (UniqueName: \"kubernetes.io/projected/87582518-d6d0-4fdb-8526-954322bbddd2-kube-api-access-hxctk\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.876326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.876227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87582518-d6d0-4fdb-8526-954322bbddd2-tls-certs\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.959889 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.959857    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:45.977241 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.977217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxctk\" (UniqueName: \"kubernetes.io/projected/87582518-d6d0-4fdb-8526-954322bbddd2-kube-api-access-hxctk\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.977319 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.977251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87582518-d6d0-4fdb-8526-954322bbddd2-tls-certs\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.979934 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.979880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87582518-d6d0-4fdb-8526-954322bbddd2-tls-certs\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:45.986920 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:45.986889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxctk\" (UniqueName: \"kubernetes.io/projected/87582518-d6d0-4fdb-8526-954322bbddd2-kube-api-access-hxctk\") pod \"model-serving-api-7b954ffc95-krzcb\" (UID: \"87582518-d6d0-4fdb-8526-954322bbddd2\") " pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:46.062003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.061916    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:46.118434 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.118402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:46.262288 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.262256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-krzcb"]
Jul 28 15:40:46.264892 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:40:46.264870    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87582518_d6d0_4fdb_8526_954322bbddd2.slice/crio-a2aa9cf82b2b4defe80b20e48573e387b03b48ad6a7e6127d31a2a43490024ff WatchSource:0}: Error finding container a2aa9cf82b2b4defe80b20e48573e387b03b48ad6a7e6127d31a2a43490024ff: Status 404 returned error can't find the container with id a2aa9cf82b2b4defe80b20e48573e387b03b48ad6a7e6127d31a2a43490024ff
Jul 28 15:40:46.266682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.266664    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:40:46.530982 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.530951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:46.639326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:46.639286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:47.163622 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:47.163586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-krzcb" event={"ID":"87582518-d6d0-4fdb-8526-954322bbddd2","Type":"ContainerStarted","Data":"a2aa9cf82b2b4defe80b20e48573e387b03b48ad6a7e6127d31a2a43490024ff"}
Jul 28 15:40:49.174345 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:49.174315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-krzcb" event={"ID":"87582518-d6d0-4fdb-8526-954322bbddd2","Type":"ContainerStarted","Data":"e227aa41de95a077621a91ad74c092c718e90dc50f8bb4efb2b6ad3907bc07d7"}
Jul 28 15:40:49.174736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:49.174419    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:40:49.195812 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:49.195754    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-krzcb" podStartSLOduration=2.056175973 podStartE2EDuration="4.195737217s" totalImagesPullingTime="2.139561244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:40:49.194481551 +0000 UTC m=+463.563134467" watchObservedRunningTime="2026-07-28 15:40:49.195737217 +0000 UTC m=+463.564390132"
Jul 28 15:40:52.302425 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:52.302386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:40:52.363340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:40:52.363308    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:00.185694 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:00.185668    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-krzcb"
Jul 28 15:41:01.876696 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:01.876662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9khhl"]
Jul 28 15:41:01.884298 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:01.884280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9khhl"
Jul 28 15:41:01.887075 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:01.887041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 15:41:01.887075 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:01.887048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7l4qw\""
Jul 28 15:41:01.887302 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:01.887201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9khhl"]
Jul 28 15:41:02.019519 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.019475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pww8w\" (UniqueName: \"kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.019670 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.019556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.120869 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.120831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pww8w\" (UniqueName: \"kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.121066 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.120882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.121285 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.121267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.129800 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.129738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pww8w\" (UniqueName: \"kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w\") pod \"s3-init-9khhl\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") " pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.210783 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.210745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9khhl"
Jul 28 15:41:02.339731 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:02.339701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9khhl"]
Jul 28 15:41:02.342866 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:41:02.342838    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ace3bec_23ac_4f63_97f9_492078ce9200.slice/crio-94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779 WatchSource:0}: Error finding container 94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779: Status 404 returned error can't find the container with id 94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779
Jul 28 15:41:03.235204 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:03.235158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerStarted","Data":"94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779"}
Jul 28 15:41:03.300513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:03.300480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:03.404684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:03.404639    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:41:07.256598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:07.256515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerStarted","Data":"0a64e9eb4fbe826601e7a67776bdd74afbd7e1bc9842efc74fffff2593b68791"}
Jul 28 15:41:09.271443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:09.271409    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerID="0a64e9eb4fbe826601e7a67776bdd74afbd7e1bc9842efc74fffff2593b68791" exitCode=0
Jul 28 15:41:09.271895 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:09.271482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerDied","Data":"0a64e9eb4fbe826601e7a67776bdd74afbd7e1bc9842efc74fffff2593b68791"}
Jul 28 15:41:13.752392 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:13.752355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 15:41:14.297968 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:41:14.297923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerStarted","Data":"afdd3fdc02870f1359c7657a544ff81cd746759de5bb02b56d8323932c7ddebc"}
Jul 28 15:42:19.557065 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:19.557034    2580 generic.go:356] "Generic (PLEG): container finished" podID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerID="afdd3fdc02870f1359c7657a544ff81cd746759de5bb02b56d8323932c7ddebc" exitCode=0
Jul 28 15:42:19.557488 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:19.557106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerDied","Data":"afdd3fdc02870f1359c7657a544ff81cd746759de5bb02b56d8323932c7ddebc"}
Jul 28 15:42:20.690639 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.690613    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9khhl"
Jul 28 15:42:20.821900 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.821805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache\") pod \"1ace3bec-23ac-4f63-97f9-492078ce9200\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") "
Jul 28 15:42:20.822075 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.821901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pww8w\" (UniqueName: \"kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w\") pod \"1ace3bec-23ac-4f63-97f9-492078ce9200\" (UID: \"1ace3bec-23ac-4f63-97f9-492078ce9200\") "
Jul 28 15:42:20.824217 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.824179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w" (OuterVolumeSpecName: "kube-api-access-pww8w") pod "1ace3bec-23ac-4f63-97f9-492078ce9200" (UID: "1ace3bec-23ac-4f63-97f9-492078ce9200"). InnerVolumeSpecName "kube-api-access-pww8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:42:20.840946 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.840912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ace3bec-23ac-4f63-97f9-492078ce9200" (UID: "1ace3bec-23ac-4f63-97f9-492078ce9200"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:42:20.923375 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.923340    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pww8w\" (UniqueName: \"kubernetes.io/projected/1ace3bec-23ac-4f63-97f9-492078ce9200-kube-api-access-pww8w\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:42:20.923375 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:20.923378    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace3bec-23ac-4f63-97f9-492078ce9200-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:42:21.566653 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:21.566618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9khhl" event={"ID":"1ace3bec-23ac-4f63-97f9-492078ce9200","Type":"ContainerDied","Data":"94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779"}
Jul 28 15:42:21.566847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:21.566668    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94b101540452b47d3c92667fc20877c42bb81c715ad7ca0678d081cc57906779"
Jul 28 15:42:21.566847 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:21.566630    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9khhl"
Jul 28 15:42:22.346887 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:22.346852    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:30.470040 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470009    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nz4md"]
Jul 28 15:42:30.470453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470381    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerName="s3-init"
Jul 28 15:42:30.470453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470393    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerName="s3-init"
Jul 28 15:42:30.470453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470412    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerName="download-hf-model"
Jul 28 15:42:30.470453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470418    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerName="download-hf-model"
Jul 28 15:42:30.470605 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.470474    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace3bec-23ac-4f63-97f9-492078ce9200" containerName="s3-init"
Jul 28 15:42:30.472714 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.472697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.475372 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.475346    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 28 15:42:30.476643 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.476625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7l4qw\""
Jul 28 15:42:30.476764 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.476630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 15:42:30.483726 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.483707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nz4md"]
Jul 28 15:42:30.609692 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.609646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1566c52-0860-4370-bb69-ae970a6f5d7f-data\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.609692 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.609705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.609921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.609756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgl6p\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-kube-api-access-qgl6p\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.711049 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.710999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1566c52-0860-4370-bb69-ae970a6f5d7f-data\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.711274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.711067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.711274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.711109    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgl6p\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-kube-api-access-qgl6p\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.711469 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.711447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1566c52-0860-4370-bb69-ae970a6f5d7f-data\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.713724 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.713708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.720907 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.720850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgl6p\" (UniqueName: \"kubernetes.io/projected/d1566c52-0860-4370-bb69-ae970a6f5d7f-kube-api-access-qgl6p\") pod \"seaweedfs-tls-custom-8b44d767d-nz4md\" (UID: \"d1566c52-0860-4370-bb69-ae970a6f5d7f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.781822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.781784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md"
Jul 28 15:42:30.915886 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:30.915861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nz4md"]
Jul 28 15:42:30.918553 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:42:30.918525    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1566c52_0860_4370_bb69_ae970a6f5d7f.slice/crio-0cebf2c55f742e173551e7cc2df746c01074cf85a4ecc5929ca34306da1f6019 WatchSource:0}: Error finding container 0cebf2c55f742e173551e7cc2df746c01074cf85a4ecc5929ca34306da1f6019: Status 404 returned error can't find the container with id 0cebf2c55f742e173551e7cc2df746c01074cf85a4ecc5929ca34306da1f6019
Jul 28 15:42:31.608022 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:31.607986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md" event={"ID":"d1566c52-0860-4370-bb69-ae970a6f5d7f","Type":"ContainerStarted","Data":"0cebf2c55f742e173551e7cc2df746c01074cf85a4ecc5929ca34306da1f6019"}
Jul 28 15:42:34.623719 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:34.623681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md" event={"ID":"d1566c52-0860-4370-bb69-ae970a6f5d7f","Type":"ContainerStarted","Data":"a4010e1f8345cc3b177adf207b8fbdb0cfc6e91d3d46b4c1b6c569b04636f68d"}
Jul 28 15:42:34.646236 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:34.646184    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-nz4md" podStartSLOduration=1.974166795 podStartE2EDuration="4.646167515s" totalImagesPullingTime="2.67200072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:42:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:42:34.643922775 +0000 UTC m=+569.012575693" watchObservedRunningTime="2026-07-28 15:42:34.646167515 +0000 UTC m=+569.014820431"
Jul 28 15:42:34.991558 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:34.991480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-w64h8"]
Jul 28 15:42:34.996203 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:34.996184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.009961 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.009922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-w64h8"]
Jul 28 15:42:35.150976 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.150927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsf2\" (UniqueName: \"kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.151229 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.150983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.252421 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.252324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsf2\" (UniqueName: \"kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.252421 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.252377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.252775 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.252753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.263462 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.263423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsf2\" (UniqueName: \"kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2\") pod \"s3-tls-init-custom-w64h8\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") " pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.315066 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.315029    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:35.426326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.426292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:35.449004 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.448981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-w64h8"]
Jul 28 15:42:35.451106 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:42:35.451081    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf8fb96f_7eea_4717_a0b7_5544121e1bdc.slice/crio-52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc WatchSource:0}: Error finding container 52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc: Status 404 returned error can't find the container with id 52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc
Jul 28 15:42:35.526246 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.526222    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:35.629962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.629912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerStarted","Data":"c97365bc6070622bd933b1fd6559a3ceb9147fa2c5ff9dbf03a6cbb60a168966"}
Jul 28 15:42:35.629962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:35.629959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerStarted","Data":"52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc"}
Jul 28 15:42:38.643067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:38.643036    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerID="c97365bc6070622bd933b1fd6559a3ceb9147fa2c5ff9dbf03a6cbb60a168966" exitCode=0
Jul 28 15:42:38.643448 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:38.643114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerDied","Data":"c97365bc6070622bd933b1fd6559a3ceb9147fa2c5ff9dbf03a6cbb60a168966"}
Jul 28 15:42:39.648956 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:39.648924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerStarted","Data":"cef47620eadd38c04b851b02db8187155a4d5f13336222743e962e75f5a0f14d"}
Jul 28 15:42:39.668659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:39.668607    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-w64h8" podStartSLOduration=3.467315287 podStartE2EDuration="5.668589168s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.201273881s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:42:39.66729868 +0000 UTC m=+574.035951598" watchObservedRunningTime="2026-07-28 15:42:39.668589168 +0000 UTC m=+574.037242085"
Jul 28 15:42:45.681369 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:45.681335    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerID="cef47620eadd38c04b851b02db8187155a4d5f13336222743e962e75f5a0f14d" exitCode=0
Jul 28 15:42:45.681756 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:45.681385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerDied","Data":"cef47620eadd38c04b851b02db8187155a4d5f13336222743e962e75f5a0f14d"}
Jul 28 15:42:46.837254 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:46.837229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:46.961728 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:46.961638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmsf2\" (UniqueName: \"kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2\") pod \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") "
Jul 28 15:42:46.961728 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:46.961707    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache\") pod \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\" (UID: \"cf8fb96f-7eea-4717-a0b7-5544121e1bdc\") "
Jul 28 15:42:46.964017 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:46.963986    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2" (OuterVolumeSpecName: "kube-api-access-gmsf2") pod "cf8fb96f-7eea-4717-a0b7-5544121e1bdc" (UID: "cf8fb96f-7eea-4717-a0b7-5544121e1bdc"). InnerVolumeSpecName "kube-api-access-gmsf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:42:47.063122 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.063081    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsf2\" (UniqueName: \"kubernetes.io/projected/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-kube-api-access-gmsf2\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:42:47.138142 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.138100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf8fb96f-7eea-4717-a0b7-5544121e1bdc" (UID: "cf8fb96f-7eea-4717-a0b7-5544121e1bdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:42:47.164085 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.164049    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf8fb96f-7eea-4717-a0b7-5544121e1bdc-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:42:47.691259 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.691217    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-w64h8" event={"ID":"cf8fb96f-7eea-4717-a0b7-5544121e1bdc","Type":"ContainerDied","Data":"52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc"}
Jul 28 15:42:47.691259 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.691264    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d3fef62b68a6a6427c44f67220ba191232274f9a9d1fab076d87f4ad8ac8bc"
Jul 28 15:42:47.691434 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:47.691235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-w64h8"
Jul 28 15:42:48.298837 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:48.298780    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:48.645329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:48.645294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:42:50.997596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.997565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-xcshw"]
Jul 28 15:42:50.998101 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.997928    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerName="s3-tls-init-custom"
Jul 28 15:42:50.998101 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.997942    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerName="s3-tls-init-custom"
Jul 28 15:42:50.998101 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.997966    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerName="download-hf-model"
Jul 28 15:42:50.998101 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.997971    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerName="download-hf-model"
Jul 28 15:42:50.998101 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:50.998050    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf8fb96f-7eea-4717-a0b7-5544121e1bdc" containerName="s3-tls-init-custom"
Jul 28 15:42:51.013596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.013573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.013768 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.013737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-xcshw"]
Jul 28 15:42:51.016729 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.016707    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 15:42:51.100055 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.100020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgjh\" (UniqueName: \"kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.100270 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.100086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.201109 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.201074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.201327 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.201171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgjh\" (UniqueName: \"kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.201522 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.201499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.211172 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.211145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgjh\" (UniqueName: \"kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh\") pod \"s3-tls-init-serving-xcshw\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") " pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.334334 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.334304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:42:51.466745 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.466710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-xcshw"]
Jul 28 15:42:51.469883 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:42:51.469853    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb44cba6c_eef1_40f8_b4e7_071a7e2449de.slice/crio-c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f WatchSource:0}: Error finding container c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f: Status 404 returned error can't find the container with id c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f
Jul 28 15:42:51.712765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.712663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerStarted","Data":"a72305dc880c1b81a5a035f104794727e5d55e33bbefff4da9589e4b8064f739"}
Jul 28 15:42:51.712765 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:51.712705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerStarted","Data":"c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f"}
Jul 28 15:42:54.727780 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:54.727734    2580 generic.go:356] "Generic (PLEG): container finished" podID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerID="a72305dc880c1b81a5a035f104794727e5d55e33bbefff4da9589e4b8064f739" exitCode=0
Jul 28 15:42:54.728201 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:54.727801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerDied","Data":"a72305dc880c1b81a5a035f104794727e5d55e33bbefff4da9589e4b8064f739"}
Jul 28 15:42:55.733565 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:55.733530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerStarted","Data":"fd77ae9735c2a2d405cbca181d543b185f027fa26ed3dcd60bd6bfe340038051"}
Jul 28 15:42:55.750674 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:42:55.750621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-xcshw" podStartSLOduration=3.362031455 podStartE2EDuration="5.750602601s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.388571146s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:42:55.750215965 +0000 UTC m=+590.118868882" watchObservedRunningTime="2026-07-28 15:42:55.750602601 +0000 UTC m=+590.119255517"
Jul 28 15:43:00.758363 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:00.758328    2580 generic.go:356] "Generic (PLEG): container finished" podID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerID="fd77ae9735c2a2d405cbca181d543b185f027fa26ed3dcd60bd6bfe340038051" exitCode=0
Jul 28 15:43:00.758741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:00.758408    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerDied","Data":"fd77ae9735c2a2d405cbca181d543b185f027fa26ed3dcd60bd6bfe340038051"}
Jul 28 15:43:01.909820 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:01.909796    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:43:02.010762 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.010728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgjh\" (UniqueName: \"kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh\") pod \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") "
Jul 28 15:43:02.010939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.010782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache\") pod \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\" (UID: \"b44cba6c-eef1-40f8-b4e7-071a7e2449de\") "
Jul 28 15:43:02.013088 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.013025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh" (OuterVolumeSpecName: "kube-api-access-5jgjh") pod "b44cba6c-eef1-40f8-b4e7-071a7e2449de" (UID: "b44cba6c-eef1-40f8-b4e7-071a7e2449de"). InnerVolumeSpecName "kube-api-access-5jgjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:43:02.034079 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.034039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache" (OuterVolumeSpecName: "model-cache") pod "b44cba6c-eef1-40f8-b4e7-071a7e2449de" (UID: "b44cba6c-eef1-40f8-b4e7-071a7e2449de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:43:02.111374 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.111342    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgjh\" (UniqueName: \"kubernetes.io/projected/b44cba6c-eef1-40f8-b4e7-071a7e2449de-kube-api-access-5jgjh\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:43:02.111374 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.111368    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b44cba6c-eef1-40f8-b4e7-071a7e2449de-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:43:02.767859 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.767834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-xcshw"
Jul 28 15:43:02.768028 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.767833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-xcshw" event={"ID":"b44cba6c-eef1-40f8-b4e7-071a7e2449de","Type":"ContainerDied","Data":"c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f"}
Jul 28 15:43:02.768028 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:02.767942    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8db45a413874782f0138566497e1c2e6642061abd127dd01f3f56aab26eab8f"
Jul 28 15:43:03.259413 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:03.259381    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:03.355421 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:03.355389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:14.914675 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:14.914643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:15.033120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033084    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:43:15.033495 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033480    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerName="s3-tls-init-serving"
Jul 28 15:43:15.033537 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033498    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerName="s3-tls-init-serving"
Jul 28 15:43:15.033537 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033520    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerName="download-hf-model"
Jul 28 15:43:15.033537 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033526    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerName="download-hf-model"
Jul 28 15:43:15.033624 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.033592    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44cba6c-eef1-40f8-b4e7-071a7e2449de" containerName="s3-tls-init-serving"
Jul 28 15:43:15.035934 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.035919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.038761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 15:43:15.038913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038739    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 15:43:15.038913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:43:15.038913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:43:15.039063 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:43:15.039063 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:43:15.039063 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.038950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:43:15.047291 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.047267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:43:15.123746 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.123708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.123960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.123773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.123960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.123796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slm5t\" (UniqueName: \"kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.123960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.123864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.123960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.123923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.224684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.224568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.224684 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.224637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.224917 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.224721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.224917 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.224773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.224917 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.224804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slm5t\" (UniqueName: \"kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.225198 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.225176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.225320 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.225303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.225361 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.225310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.227308 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.227289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.233665 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.233642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slm5t\" (UniqueName: \"kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.347274 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.347245    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:15.485939 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.485910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:43:15.487774 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:43:15.487733    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4861b1_c40d_44fa_ba2f_126562c48b06.slice/crio-722faa04107e69f9b55e9a23a8308f012bf8405897a266763919dfdec355d4a2 WatchSource:0}: Error finding container 722faa04107e69f9b55e9a23a8308f012bf8405897a266763919dfdec355d4a2: Status 404 returned error can't find the container with id 722faa04107e69f9b55e9a23a8308f012bf8405897a266763919dfdec355d4a2
Jul 28 15:43:15.824698 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.824604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerStarted","Data":"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"}
Jul 28 15:43:15.824842 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:15.824740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerStarted","Data":"722faa04107e69f9b55e9a23a8308f012bf8405897a266763919dfdec355d4a2"}
Jul 28 15:43:17.142393 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:17.142357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:43:19.842471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:19.842438    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerID="21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab" exitCode=0
Jul 28 15:43:19.842834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:19.842514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerDied","Data":"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"}
Jul 28 15:43:32.919631 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:32.919597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerStarted","Data":"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"}
Jul 28 15:43:35.940099 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:35.940055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerStarted","Data":"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"}
Jul 28 15:43:37.952007 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:37.951922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerStarted","Data":"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"}
Jul 28 15:43:37.952431 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:37.952158    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:37.952431 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:37.952206    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:37.953672 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:37.953635    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:43:37.976773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:37.976683    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podStartSLOduration=1.469055642 podStartE2EDuration="22.976656746s" totalImagesPullingTime="17.628050479s" totalInitContainerRuntime="3.879550625s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:43:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:43:37.973185469 +0000 UTC m=+632.341838388" watchObservedRunningTime="2026-07-28 15:43:37.976656746 +0000 UTC m=+632.345309661"
Jul 28 15:43:38.956315 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:38.956281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:38.956876 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:38.956493    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:43:38.957746 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:38.957716    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:43:39.960654 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:39.960611    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:43:39.961058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:39.961007    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:43:44.966567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:44.966539    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:43:44.967300 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:44.967265    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:43:44.967523 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:44.967499    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:43:54.968145 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:54.968087    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:43:54.968738 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:43:54.968621    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:04.967465 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:04.967421    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:44:04.967999 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:04.967945    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:14.967464 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:14.967424    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:44:14.967969 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:14.967942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:24.967471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:24.967427    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:44:24.968028 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:24.967872    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:34.967329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:34.967288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:44:34.967856 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:34.967830    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:44.968017 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:44.967974    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:44:44.968569 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:44.968397    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:44:54.968061 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:54.968028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:44:54.968600 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:44:54.968210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:10.155606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.155569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:10.203031 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.202996    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:45:10.206057 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.206034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.207430 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.207410    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:45:10.207683 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.207646    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" containerID="cri-o://358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b" gracePeriod=30
Jul 28 15:45:10.207800 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.207667    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" containerID="cri-o://999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3" gracePeriod=30
Jul 28 15:45:10.207800 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.207657    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" containerID="cri-o://2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f" gracePeriod=30
Jul 28 15:45:10.208688 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.208394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 15:45:10.208688 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.208402    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 15:45:10.217595 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.217572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:45:10.241281 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.241254    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:10.356524 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.356494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.356524 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.356529    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.356778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.356557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.356778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.356646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwn5\" (UniqueName: \"kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.356778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.356687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.362348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.362319    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerID="999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3" exitCode=2
Jul 28 15:45:10.362499 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.362404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerDied","Data":"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"}
Jul 28 15:45:10.457497 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.457410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.457497 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.457453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.457497 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.457472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.457497 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.457496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwn5\" (UniqueName: \"kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.457874 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.457516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.457874 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:10.457572    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 28 15:45:10.457874 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:10.457651    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls podName:b93f9fbd-4a11-41ae-8965-0c21b3c380e1 nodeName:}" failed. No retries permitted until 2026-07-28 15:45:10.957627776 +0000 UTC m=+725.326280670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 28 15:45:10.458219 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.458200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.458329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.458271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.458329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.458294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.467809 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.467785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwn5\" (UniqueName: \"kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.602312 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.602281    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:10.963511 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.963458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:10.966248 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:10.966215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:11.126270 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:11.126237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:11.265341 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:11.265312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:45:11.267013 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:45:11.266982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93f9fbd_4a11_41ae_8965_0c21b3c380e1.slice/crio-f1163082f40c1d6099900d7425e897111359e9db2d73e2a1b4ab0cdee3414e58 WatchSource:0}: Error finding container f1163082f40c1d6099900d7425e897111359e9db2d73e2a1b4ab0cdee3414e58: Status 404 returned error can't find the container with id f1163082f40c1d6099900d7425e897111359e9db2d73e2a1b4ab0cdee3414e58
Jul 28 15:45:11.369514 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:11.369473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerStarted","Data":"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"}
Jul 28 15:45:11.369703 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:11.369547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerStarted","Data":"f1163082f40c1d6099900d7425e897111359e9db2d73e2a1b4ab0cdee3414e58"}
Jul 28 15:45:14.961355 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:14.961309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:14.968017 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:14.967978    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:45:14.968319 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:14.968291    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:15.390166 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:15.390118    2580 generic.go:356] "Generic (PLEG): container finished" podID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerID="f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47" exitCode=0
Jul 28 15:45:15.390166 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:15.390158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerDied","Data":"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"}
Jul 28 15:45:15.392964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:15.392942    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerID="358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b" exitCode=0
Jul 28 15:45:15.393045 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:15.393010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerDied","Data":"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"}
Jul 28 15:45:16.399841 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.399808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerStarted","Data":"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"}
Jul 28 15:45:16.400268 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.399884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerStarted","Data":"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"}
Jul 28 15:45:16.400268 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.399910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerStarted","Data":"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"}
Jul 28 15:45:16.400268 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.400216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:16.400402 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.400278    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:16.400402 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.400302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:16.401741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.401714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:16.402393 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.402358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:16.424696 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:16.424650    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podStartSLOduration=2.4023824019999998 podStartE2EDuration="6.424635051s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.022252649s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:45:16.422954476 +0000 UTC m=+730.791607396" watchObservedRunningTime="2026-07-28 15:45:16.424635051 +0000 UTC m=+730.793288036"
Jul 28 15:45:17.404700 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:17.404651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:17.405084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:17.405060    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:19.961714 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:19.961673    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:22.409429 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:22.409400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:45:22.409984 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:22.409952    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:22.410322 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:22.410294    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:24.961593 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:24.961545    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:24.961995 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:24.961694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:24.967613 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:24.967579    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:45:24.967920 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:24.967887    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:29.960763 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:29.960721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:32.410043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:32.410000    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:32.410517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:32.410488    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:32.956317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:32.956287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:45:34.961744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:34.961704    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:34.967250 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:34.967201    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 15:45:34.967428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:34.967411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:34.967580 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:34.967561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:34.967657 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:34.967646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:39.961010 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:39.960966    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 15:45:40.359190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.359164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:40.419605 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.419577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config\") pod \"fa4861b1-c40d-44fa-ba2f-126562c48b06\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") "
Jul 28 15:45:40.419778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.419642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slm5t\" (UniqueName: \"kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t\") pod \"fa4861b1-c40d-44fa-ba2f-126562c48b06\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") "
Jul 28 15:45:40.419778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.419715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls\") pod \"fa4861b1-c40d-44fa-ba2f-126562c48b06\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") "
Jul 28 15:45:40.419778 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.419756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert\") pod \"fa4861b1-c40d-44fa-ba2f-126562c48b06\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") "
Jul 28 15:45:40.419919 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.419807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location\") pod \"fa4861b1-c40d-44fa-ba2f-126562c48b06\" (UID: \"fa4861b1-c40d-44fa-ba2f-126562c48b06\") "
Jul 28 15:45:40.420149 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.420003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa4861b1-c40d-44fa-ba2f-126562c48b06" (UID: "fa4861b1-c40d-44fa-ba2f-126562c48b06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:40.420298 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.420210    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa4861b1-c40d-44fa-ba2f-126562c48b06" (UID: "fa4861b1-c40d-44fa-ba2f-126562c48b06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:45:40.420298 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.420214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa4861b1-c40d-44fa-ba2f-126562c48b06" (UID: "fa4861b1-c40d-44fa-ba2f-126562c48b06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:45:40.422036 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.422009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t" (OuterVolumeSpecName: "kube-api-access-slm5t") pod "fa4861b1-c40d-44fa-ba2f-126562c48b06" (UID: "fa4861b1-c40d-44fa-ba2f-126562c48b06"). InnerVolumeSpecName "kube-api-access-slm5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:45:40.422036 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.422014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa4861b1-c40d-44fa-ba2f-126562c48b06" (UID: "fa4861b1-c40d-44fa-ba2f-126562c48b06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:45:40.516212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.516101    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerID="2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f" exitCode=0
Jul 28 15:45:40.516212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.516156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerDied","Data":"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"}
Jul 28 15:45:40.516212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.516203    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"
Jul 28 15:45:40.516473 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.516221    2580 scope.go:117] "RemoveContainer" containerID="2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"
Jul 28 15:45:40.516473 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.516211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg" event={"ID":"fa4861b1-c40d-44fa-ba2f-126562c48b06","Type":"ContainerDied","Data":"722faa04107e69f9b55e9a23a8308f012bf8405897a266763919dfdec355d4a2"}
Jul 28 15:45:40.520467 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.520445    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:45:40.520467 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.520467    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slm5t\" (UniqueName: \"kubernetes.io/projected/fa4861b1-c40d-44fa-ba2f-126562c48b06-kube-api-access-slm5t\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:45:40.520630 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.520476    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa4861b1-c40d-44fa-ba2f-126562c48b06-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:45:40.520630 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.520486    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa4861b1-c40d-44fa-ba2f-126562c48b06-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:45:40.520630 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.520493    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa4861b1-c40d-44fa-ba2f-126562c48b06-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:45:40.525298 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.525280    2580 scope.go:117] "RemoveContainer" containerID="999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"
Jul 28 15:45:40.535427 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.535333    2580 scope.go:117] "RemoveContainer" containerID="358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"
Jul 28 15:45:40.543616 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.543596    2580 scope.go:117] "RemoveContainer" containerID="21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"
Jul 28 15:45:40.547359 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.547321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:45:40.549397 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.549378    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-2llrg"]
Jul 28 15:45:40.552457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.552439    2580 scope.go:117] "RemoveContainer" containerID="2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"
Jul 28 15:45:40.552716 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:40.552698    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f\": container with ID starting with 2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f not found: ID does not exist" containerID="2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"
Jul 28 15:45:40.552775 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.552725    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f"} err="failed to get container status \"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f\": rpc error: code = NotFound desc = could not find container \"2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f\": container with ID starting with 2523aa2ead9c26ce1f7160dcfdc12b981f7bba743c59030ba5e9506eedf5879f not found: ID does not exist"
Jul 28 15:45:40.552775 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.552745    2580 scope.go:117] "RemoveContainer" containerID="999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"
Jul 28 15:45:40.552976 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:40.552962    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3\": container with ID starting with 999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3 not found: ID does not exist" containerID="999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"
Jul 28 15:45:40.553011 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.552979    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3"} err="failed to get container status \"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3\": rpc error: code = NotFound desc = could not find container \"999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3\": container with ID starting with 999291611e47a1fef9006002f6351cd785cefc886585bdee4d7a0a06b3f956d3 not found: ID does not exist"
Jul 28 15:45:40.553011 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.552996    2580 scope.go:117] "RemoveContainer" containerID="358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"
Jul 28 15:45:40.553218 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:40.553201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b\": container with ID starting with 358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b not found: ID does not exist" containerID="358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"
Jul 28 15:45:40.553256 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.553222    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b"} err="failed to get container status \"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b\": rpc error: code = NotFound desc = could not find container \"358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b\": container with ID starting with 358cce0244c9fb60f3675a27dd7d4667fec6476d691ceb9ac7644190b4f88c1b not found: ID does not exist"
Jul 28 15:45:40.553256 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.553235    2580 scope.go:117] "RemoveContainer" containerID="21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"
Jul 28 15:45:40.553421 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:45:40.553404    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab\": container with ID starting with 21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab not found: ID does not exist" containerID="21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"
Jul 28 15:45:40.553466 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:40.553427    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab"} err="failed to get container status \"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab\": rpc error: code = NotFound desc = could not find container \"21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab\": container with ID starting with 21452c7c465a6a0b74d04ce692327a2239e0bb2eaf31418cbf5036c73a5780ab not found: ID does not exist"
Jul 28 15:45:42.186513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:42.186481    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" path="/var/lib/kubelet/pods/fa4861b1-c40d-44fa-ba2f-126562c48b06/volumes"
Jul 28 15:45:42.410076 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:42.410035    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:42.410543 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:42.410519    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:45:52.410290 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:52.410246    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:45:52.410737 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:45:52.410714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:46:02.410144 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:02.410082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:46:02.410613 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:02.410590    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:46:12.410662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:12.410618    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:46:12.411100 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:12.411078    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:46:22.410785 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:22.410744    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:46:22.411296 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:22.411258    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:46:32.411007 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:32.410918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:46:32.411395 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:32.411121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:46:45.527112 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.527080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:45.646793 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.646759    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:46:45.647291 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647274    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container"
Jul 28 15:46:45.647291 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647293    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647303    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647310    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647320    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647328    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647354    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="storage-initializer"
Jul 28 15:46:45.647383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647359    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="storage-initializer"
Jul 28 15:46:45.647561 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647411    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kserve-container"
Jul 28 15:46:45.647561 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647420    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="kube-rbac-proxy"
Jul 28 15:46:45.647561 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.647428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4861b1-c40d-44fa-ba2f-126562c48b06" containerName="agent"
Jul 28 15:46:45.649505 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.649491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.655897 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.655870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 15:46:45.656068 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.655917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 15:46:45.669975 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.669945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:46:45.694611 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.694575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:46:45.694874 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.694845    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" containerID="cri-o://a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6" gracePeriod=30
Jul 28 15:46:45.694951 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.694896    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" containerID="cri-o://c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353" gracePeriod=30
Jul 28 15:46:45.694991 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.694834    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" containerID="cri-o://7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3" gracePeriod=30
Jul 28 15:46:45.705972 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.705945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:45.806261 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.806178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxtq7\" (UniqueName: \"kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.806415 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.806277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.806415 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.806314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.907697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.907795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.907882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxtq7\" (UniqueName: \"kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:46:45.908474    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:46:45.908554    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls podName:5a632ca1-e89c-4462-bba0-69ae4efbe47e nodeName:}" failed. No retries permitted until 2026-07-28 15:46:46.408532933 +0000 UTC m=+820.777185831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-s2dzj" (UID: "5a632ca1-e89c-4462-bba0-69ae4efbe47e") : secret "message-dumper-predictor-serving-cert" not found
Jul 28 15:46:45.912160 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.909466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:45.923548 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:45.923515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxtq7\" (UniqueName: \"kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:46.033309 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.033279    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:46:46.412506 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.412462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:46.415038 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.415019    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-s2dzj\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:46.560246 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.560194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:46.702209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.702186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:46:46.704492 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:46:46.704464    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a632ca1_e89c_4462_bba0_69ae4efbe47e.slice/crio-8d294bb207de669dbb85e632dbace92d47510a89af31b7f445d1e7d586f66137 WatchSource:0}: Error finding container 8d294bb207de669dbb85e632dbace92d47510a89af31b7f445d1e7d586f66137: Status 404 returned error can't find the container with id 8d294bb207de669dbb85e632dbace92d47510a89af31b7f445d1e7d586f66137
Jul 28 15:46:46.706829 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.706809    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 15:46:46.808659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.808619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerStarted","Data":"8d294bb207de669dbb85e632dbace92d47510a89af31b7f445d1e7d586f66137"}
Jul 28 15:46:46.811152 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.811108    2580 generic.go:356] "Generic (PLEG): container finished" podID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerID="c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353" exitCode=2
Jul 28 15:46:46.811269 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:46.811163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerDied","Data":"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"}
Jul 28 15:46:47.405138 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:47.405077    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:46:48.824085 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:48.824050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerStarted","Data":"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"}
Jul 28 15:46:48.824506 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:48.824107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerStarted","Data":"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"}
Jul 28 15:46:48.824506 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:48.824172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:48.850144 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:48.850069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" podStartSLOduration=2.570435662 podStartE2EDuration="3.850053396s" totalImagesPullingTime="1.279617734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:46:48.846840019 +0000 UTC m=+823.215492936" watchObservedRunningTime="2026-07-28 15:46:48.850053396 +0000 UTC m=+823.218706312"
Jul 28 15:46:49.829424 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:49.829393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:49.831182 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:49.831161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:50.836354 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:50.836327    2580 generic.go:356] "Generic (PLEG): container finished" podID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerID="7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3" exitCode=0
Jul 28 15:46:50.836714 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:50.836400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerDied","Data":"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"}
Jul 28 15:46:52.405766 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:52.405718    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:46:52.410148 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:52.410096    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:46:52.410479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:52.410454    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:46:56.845906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:56.845873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:46:57.405681 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:57.405647    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:46:57.405842 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:46:57.405779    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:47:02.405085 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:02.405041    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:47:02.410483 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:02.410450    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:47:02.410811 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:02.410785    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:05.515238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.515196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:05.592649 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.588683    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:47:05.594795 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.594769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.597834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.597815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 15:47:05.598250 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.598230    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 15:47:05.604189 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.604167    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:47:05.684596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.684549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.684776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.684624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.684776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.684656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.684776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.684673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nt9\" (UniqueName: \"kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.684776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.684704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786048 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.785963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786048 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786318 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786318 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nt9\" (UniqueName: \"kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786318 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786318 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:05.786277    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 28 15:47:05.786524 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:05.786369    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls podName:2b25c6fd-eff1-4aa7-93a9-0b4afea04b12 nodeName:}" failed. No retries permitted until 2026-07-28 15:47:06.28634699 +0000 UTC m=+840.654999886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls") pod "isvc-logger-predictor-6b6746bb7-5rwvr" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12") : secret "isvc-logger-predictor-serving-cert" not found
Jul 28 15:47:05.786611 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786851 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.786891 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.786831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.796425 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.796400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nt9\" (UniqueName: \"kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:05.885977 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:05.885948    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:06.290843 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.290809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:06.293346 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.293317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-5rwvr\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:06.507231 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.507192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:06.655267 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.655199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:47:06.657432 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:47:06.657405    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b25c6fd_eff1_4aa7_93a9_0b4afea04b12.slice/crio-abeea204a703503c2c01f0f30f73a587ffe72e8995100c4b497baf6b907ee4f7 WatchSource:0}: Error finding container abeea204a703503c2c01f0f30f73a587ffe72e8995100c4b497baf6b907ee4f7: Status 404 returned error can't find the container with id abeea204a703503c2c01f0f30f73a587ffe72e8995100c4b497baf6b907ee4f7
Jul 28 15:47:06.911445 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.911411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerStarted","Data":"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"}
Jul 28 15:47:06.911609 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:06.911471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerStarted","Data":"abeea204a703503c2c01f0f30f73a587ffe72e8995100c4b497baf6b907ee4f7"}
Jul 28 15:47:07.405293 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:07.405254    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:47:10.932635 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:10.932603    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerID="7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf" exitCode=0
Jul 28 15:47:10.933013 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:10.932684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerDied","Data":"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"}
Jul 28 15:47:11.940302 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.940263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerStarted","Data":"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"}
Jul 28 15:47:11.940741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.940339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerStarted","Data":"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"}
Jul 28 15:47:11.940741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.940373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerStarted","Data":"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"}
Jul 28 15:47:11.940741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.940656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:11.940906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.940818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:11.942216 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.942189    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:11.964626 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:11.964582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podStartSLOduration=3.336139182 podStartE2EDuration="6.964568609s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.628429427s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:47:11.963709737 +0000 UTC m=+846.332362664" watchObservedRunningTime="2026-07-28 15:47:11.964568609 +0000 UTC m=+846.333221599"
Jul 28 15:47:12.405549 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.405512    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 15:47:12.409937 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.409906    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
Jul 28 15:47:12.410038 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.410024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:47:12.410114 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.410096    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:12.410225 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.410211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:47:12.945708 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.945665    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:12.945708 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.945696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:12.946940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:12.946913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:13.951348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:13.951306    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:13.951828 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:13.951798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:15.845018 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.844958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:47:15.962491 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.962404    2580 generic.go:356] "Generic (PLEG): container finished" podID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerID="a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6" exitCode=0
Jul 28 15:47:15.962491 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.962475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerDied","Data":"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"}
Jul 28 15:47:15.962666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.962487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"
Jul 28 15:47:15.962666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.962565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n" event={"ID":"b93f9fbd-4a11-41ae-8965-0c21b3c380e1","Type":"ContainerDied","Data":"f1163082f40c1d6099900d7425e897111359e9db2d73e2a1b4ab0cdee3414e58"}
Jul 28 15:47:15.962666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.962599    2580 scope.go:117] "RemoveContainer" containerID="a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"
Jul 28 15:47:15.971160 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.971116    2580 scope.go:117] "RemoveContainer" containerID="c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"
Jul 28 15:47:15.979398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.979378    2580 scope.go:117] "RemoveContainer" containerID="7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"
Jul 28 15:47:15.982623 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location\") pod \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") "
Jul 28 15:47:15.982716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") pod \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") "
Jul 28 15:47:15.982716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config\") pod \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") "
Jul 28 15:47:15.982716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jwn5\" (UniqueName: \"kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5\") pod \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") "
Jul 28 15:47:15.982866 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert\") pod \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\" (UID: \"b93f9fbd-4a11-41ae-8965-0c21b3c380e1\") "
Jul 28 15:47:15.983015 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.982925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b93f9fbd-4a11-41ae-8965-0c21b3c380e1" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:47:15.983096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.983075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b93f9fbd-4a11-41ae-8965-0c21b3c380e1" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:47:15.983264 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.983150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b93f9fbd-4a11-41ae-8965-0c21b3c380e1" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:47:15.985034 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.985003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5" (OuterVolumeSpecName: "kube-api-access-2jwn5") pod "b93f9fbd-4a11-41ae-8965-0c21b3c380e1" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1"). InnerVolumeSpecName "kube-api-access-2jwn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:47:15.985143 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.985010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b93f9fbd-4a11-41ae-8965-0c21b3c380e1" (UID: "b93f9fbd-4a11-41ae-8965-0c21b3c380e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:47:15.987742 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:15.987725    2580 scope.go:117] "RemoveContainer" containerID="f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"
Jul 28 15:47:16.000043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000023    2580 scope.go:117] "RemoveContainer" containerID="a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"
Jul 28 15:47:16.000335 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:16.000314    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6\": container with ID starting with a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6 not found: ID does not exist" containerID="a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"
Jul 28 15:47:16.000418 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000349    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6"} err="failed to get container status \"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6\": rpc error: code = NotFound desc = could not find container \"a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6\": container with ID starting with a9d65aef5238f5b7d4a3225632a28b8d07c522193274202e1e0d4147fe56e3e6 not found: ID does not exist"
Jul 28 15:47:16.000418 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000377    2580 scope.go:117] "RemoveContainer" containerID="c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"
Jul 28 15:47:16.000620 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:16.000604    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353\": container with ID starting with c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353 not found: ID does not exist" containerID="c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"
Jul 28 15:47:16.000658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000627    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353"} err="failed to get container status \"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353\": rpc error: code = NotFound desc = could not find container \"c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353\": container with ID starting with c8f2f1f8fe367c3e59b0a018d147325704f35b687556af4e7535881af54d8353 not found: ID does not exist"
Jul 28 15:47:16.000658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000642    2580 scope.go:117] "RemoveContainer" containerID="7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"
Jul 28 15:47:16.000893 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:16.000876    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3\": container with ID starting with 7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3 not found: ID does not exist" containerID="7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"
Jul 28 15:47:16.000931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000897    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3"} err="failed to get container status \"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3\": rpc error: code = NotFound desc = could not find container \"7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3\": container with ID starting with 7527d511b1c638a1cd25a8275cfc3a651f39b921deacf167320310d4376e0ff3 not found: ID does not exist"
Jul 28 15:47:16.000931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.000910    2580 scope.go:117] "RemoveContainer" containerID="f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"
Jul 28 15:47:16.001147 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:47:16.001110    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47\": container with ID starting with f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47 not found: ID does not exist" containerID="f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"
Jul 28 15:47:16.001198 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.001156    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47"} err="failed to get container status \"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47\": rpc error: code = NotFound desc = could not find container \"f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47\": container with ID starting with f786c1a9d58127f76d4c2ad7812379119501f6c20daa9ea67f8b0208d0805d47 not found: ID does not exist"
Jul 28 15:47:16.084296 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.084258    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:47:16.084296 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.084288    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:47:16.084296 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.084298    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:47:16.084296 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.084308    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwn5\" (UniqueName: \"kubernetes.io/projected/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-kube-api-access-2jwn5\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:47:16.084564 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.084317    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b93f9fbd-4a11-41ae-8965-0c21b3c380e1-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:47:16.282664 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.282584    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:47:16.286979 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:16.286953    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-7rm6n"]
Jul 28 15:47:18.186314 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:18.186270    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" path="/var/lib/kubelet/pods/b93f9fbd-4a11-41ae-8965-0c21b3c380e1/volumes"
Jul 28 15:47:18.955633 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:18.955606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:47:18.956169 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:18.956119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:18.956550 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:18.956526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:19.819218 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:19.819186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:47:28.956587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:28.956541    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:28.957085 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:28.956915    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:38.956086 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:38.956047    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:38.956573 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:38.956547    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:48.956504 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:48.956463    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:48.956994 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:48.956893    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:47:58.956641 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:58.956544    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:47:58.957113 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:47:58.957087    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:48:08.956122 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:08.956082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:48:08.956622 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:08.956516    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:48:18.956082 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:18.956033    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:48:18.958414 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:18.956503    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:48:28.957146 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:28.957108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:48:28.957533 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:28.957241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:48:40.540640 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.540610    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-s2dzj_5a632ca1-e89c-4462-bba0-69ae4efbe47e/kserve-container/0.log"
Jul 28 15:48:40.784043 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.784007    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:48:40.784376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.784323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" containerID="cri-o://29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a" gracePeriod=30
Jul 28 15:48:40.784489 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.784416    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" containerID="cri-o://48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b" gracePeriod=30
Jul 28 15:48:40.784489 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.784353    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" containerID="cri-o://9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709" gracePeriod=30
Jul 28 15:48:40.806642 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.806613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:48:40.858504 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858472    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:48:40.858889 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="storage-initializer"
Jul 28 15:48:40.858931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858892    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="storage-initializer"
Jul 28 15:48:40.858931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858906    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy"
Jul 28 15:48:40.858931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy"
Jul 28 15:48:40.858931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858924    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container"
Jul 28 15:48:40.858931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container"
Jul 28 15:48:40.859084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858937    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent"
Jul 28 15:48:40.859084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858942    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent"
Jul 28 15:48:40.859084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.858992    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kserve-container"
Jul 28 15:48:40.859084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.859000    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="kube-rbac-proxy"
Jul 28 15:48:40.859084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.859006    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93f9fbd-4a11-41ae-8965-0c21b3c380e1" containerName="agent"
Jul 28 15:48:40.862669 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.862649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:40.865239 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.865220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 15:48:40.865340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.865273    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 15:48:40.873653 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.873630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:48:40.912113 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.912082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:48:40.912373 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.912333    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kserve-container" containerID="cri-o://5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20" gracePeriod=30
Jul 28 15:48:40.912463 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.912413    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kube-rbac-proxy" containerID="cri-o://f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40" gracePeriod=30
Jul 28 15:48:40.919914 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.919892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:40.920003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.919934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:40.920003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.920001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjddv\" (UniqueName: \"kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:40.920084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.920027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:40.920084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:40.920047    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.020657 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.020625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.020822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.020682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.020822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.020770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjddv\" (UniqueName: \"kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.020822 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.020811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.021067 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.020995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.021216 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.021185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.021443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.021422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.021732 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.021708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.023216 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.023183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.031537 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.031519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjddv\" (UniqueName: \"kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv\") pod \"isvc-autogluon-v1-predictor-574b5b7495-ztc27\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.151900 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.151875    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:48:41.172921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.172854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:48:41.224284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.223257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config\") pod \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") "
Jul 28 15:48:41.224284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.223351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxtq7\" (UniqueName: \"kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7\") pod \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") "
Jul 28 15:48:41.224284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.223423    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") pod \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\" (UID: \"5a632ca1-e89c-4462-bba0-69ae4efbe47e\") "
Jul 28 15:48:41.224284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.223681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a632ca1-e89c-4462-bba0-69ae4efbe47e" (UID: "5a632ca1-e89c-4462-bba0-69ae4efbe47e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:48:41.224606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.224401    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:48:41.226832 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.226807    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7" (OuterVolumeSpecName: "kube-api-access-gxtq7") pod "5a632ca1-e89c-4462-bba0-69ae4efbe47e" (UID: "5a632ca1-e89c-4462-bba0-69ae4efbe47e"). InnerVolumeSpecName "kube-api-access-gxtq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:48:41.226959 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.226857    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a632ca1-e89c-4462-bba0-69ae4efbe47e" (UID: "5a632ca1-e89c-4462-bba0-69ae4efbe47e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:48:41.311775 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.311740    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:48:41.314003 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:48:41.313978    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd4392a_32f8_471a_85a8_cb934b67a3c2.slice/crio-a7684925392464460ef2db49136aceb410b1928b920ba3988b49b1026772b33e WatchSource:0}: Error finding container a7684925392464460ef2db49136aceb410b1928b920ba3988b49b1026772b33e: Status 404 returned error can't find the container with id a7684925392464460ef2db49136aceb410b1928b920ba3988b49b1026772b33e
Jul 28 15:48:41.325199 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.325178    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxtq7\" (UniqueName: \"kubernetes.io/projected/5a632ca1-e89c-4462-bba0-69ae4efbe47e-kube-api-access-gxtq7\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:48:41.325199 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.325200    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a632ca1-e89c-4462-bba0-69ae4efbe47e-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:48:41.375948 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.375923    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerID="48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b" exitCode=2
Jul 28 15:48:41.376039 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.375976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerDied","Data":"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"}
Jul 28 15:48:41.377620 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377595    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerID="f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40" exitCode=2
Jul 28 15:48:41.377736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377622    2580 generic.go:356] "Generic (PLEG): container finished" podID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerID="5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20" exitCode=2
Jul 28 15:48:41.377736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377640    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"
Jul 28 15:48:41.377736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerDied","Data":"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"}
Jul 28 15:48:41.377736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerDied","Data":"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"}
Jul 28 15:48:41.377936 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj" event={"ID":"5a632ca1-e89c-4462-bba0-69ae4efbe47e","Type":"ContainerDied","Data":"8d294bb207de669dbb85e632dbace92d47510a89af31b7f445d1e7d586f66137"}
Jul 28 15:48:41.377936 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.377789    2580 scope.go:117] "RemoveContainer" containerID="f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"
Jul 28 15:48:41.379647 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.379626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerStarted","Data":"a7684925392464460ef2db49136aceb410b1928b920ba3988b49b1026772b33e"}
Jul 28 15:48:41.387119 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.387096    2580 scope.go:117] "RemoveContainer" containerID="5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"
Jul 28 15:48:41.395071 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395052    2580 scope.go:117] "RemoveContainer" containerID="f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"
Jul 28 15:48:41.395413 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:48:41.395395    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40\": container with ID starting with f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40 not found: ID does not exist" containerID="f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"
Jul 28 15:48:41.395472 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395421    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"} err="failed to get container status \"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40\": rpc error: code = NotFound desc = could not find container \"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40\": container with ID starting with f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40 not found: ID does not exist"
Jul 28 15:48:41.395472 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395443    2580 scope.go:117] "RemoveContainer" containerID="5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"
Jul 28 15:48:41.395712 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:48:41.395694    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20\": container with ID starting with 5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20 not found: ID does not exist" containerID="5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"
Jul 28 15:48:41.395779 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395722    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"} err="failed to get container status \"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20\": rpc error: code = NotFound desc = could not find container \"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20\": container with ID starting with 5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20 not found: ID does not exist"
Jul 28 15:48:41.395779 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395747    2580 scope.go:117] "RemoveContainer" containerID="f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"
Jul 28 15:48:41.396000 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.395982    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40"} err="failed to get container status \"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40\": rpc error: code = NotFound desc = could not find container \"f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40\": container with ID starting with f2553ee7cc6dcbcb1c02440d0382b89cdde6118107fb4389153f7da12ed03f40 not found: ID does not exist"
Jul 28 15:48:41.396078 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.396002    2580 scope.go:117] "RemoveContainer" containerID="5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"
Jul 28 15:48:41.396317 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.396299    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20"} err="failed to get container status \"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20\": rpc error: code = NotFound desc = could not find container \"5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20\": container with ID starting with 5751441500d15bd60648a3d98916cfdedf694cc343246e10459eacedd9368e20 not found: ID does not exist"
Jul 28 15:48:41.426662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.426633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:48:41.433563 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:41.433538    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-s2dzj"]
Jul 28 15:48:42.174954 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:42.174919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:48:42.186486 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:42.186457    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" path="/var/lib/kubelet/pods/5a632ca1-e89c-4462-bba0-69ae4efbe47e/volumes"
Jul 28 15:48:42.387475 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:42.387444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerStarted","Data":"e3adae6ffad911284a1c9b9520df1c450811d3efa77daedde1824cf9227d70a9"}
Jul 28 15:48:43.952082 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:43.952041    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:48:45.403753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:45.403727    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerID="29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a" exitCode=0
Jul 28 15:48:45.404064 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:45.403799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerDied","Data":"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"}
Jul 28 15:48:48.951728 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:48.951685    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:48:48.956574 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:48.956537    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:48:48.956895 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:48.956874    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:48:53.951719 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:53.951667    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:48:53.952232 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:53.951836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:48:58.951716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:58.951663    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:48:58.956207 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:58.956175    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:48:58.956535 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:48:58.956508    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:03.952348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:03.952301    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:49:08.952024 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:08.951982    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 15:49:08.956489 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:08.956457    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 15:49:08.956644 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:08.956579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:49:08.956753 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:08.956730    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 15:49:08.956861 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:08.956850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:49:10.933156 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:10.933111    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:49:11.082801 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.082771    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8nt9\" (UniqueName: \"kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9\") pod \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") "
Jul 28 15:49:11.082964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.082833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location\") pod \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") "
Jul 28 15:49:11.082964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.082873    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config\") pod \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") "
Jul 28 15:49:11.082964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.082897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert\") pod \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") "
Jul 28 15:49:11.082964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.082929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") pod \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\" (UID: \"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12\") "
Jul 28 15:49:11.083246 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.083221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:49:11.083313 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.083271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:49:11.083313 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.083281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:49:11.085120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.085095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:49:11.085120 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.085093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9" (OuterVolumeSpecName: "kube-api-access-n8nt9") pod "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" (UID: "2b25c6fd-eff1-4aa7-93a9-0b4afea04b12"). InnerVolumeSpecName "kube-api-access-n8nt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:49:11.184247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.184173    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:49:11.184247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.184198    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:49:11.184247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.184208    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:49:11.184247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.184217    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:49:11.184247 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.184225    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nt9\" (UniqueName: \"kubernetes.io/projected/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12-kube-api-access-n8nt9\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:49:11.527834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.527746    2580 generic.go:356] "Generic (PLEG): container finished" podID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerID="e3adae6ffad911284a1c9b9520df1c450811d3efa77daedde1824cf9227d70a9" exitCode=0
Jul 28 15:49:11.527834 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.527821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerDied","Data":"e3adae6ffad911284a1c9b9520df1c450811d3efa77daedde1824cf9227d70a9"}
Jul 28 15:49:11.530254 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.530231    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerID="9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709" exitCode=0
Jul 28 15:49:11.530340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.530270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerDied","Data":"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"}
Jul 28 15:49:11.530340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.530294    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"
Jul 28 15:49:11.530340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.530324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr" event={"ID":"2b25c6fd-eff1-4aa7-93a9-0b4afea04b12","Type":"ContainerDied","Data":"abeea204a703503c2c01f0f30f73a587ffe72e8995100c4b497baf6b907ee4f7"}
Jul 28 15:49:11.530436 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.530355    2580 scope.go:117] "RemoveContainer" containerID="9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"
Jul 28 15:49:11.539284 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.539264    2580 scope.go:117] "RemoveContainer" containerID="48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"
Jul 28 15:49:11.546900 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.546880    2580 scope.go:117] "RemoveContainer" containerID="29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"
Jul 28 15:49:11.555518 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.555506    2580 scope.go:117] "RemoveContainer" containerID="7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"
Jul 28 15:49:11.563081 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563066    2580 scope.go:117] "RemoveContainer" containerID="9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"
Jul 28 15:49:11.563335 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:49:11.563319    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709\": container with ID starting with 9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709 not found: ID does not exist" containerID="9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"
Jul 28 15:49:11.563385 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563343    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709"} err="failed to get container status \"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709\": rpc error: code = NotFound desc = could not find container \"9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709\": container with ID starting with 9873f3ef6764b5e75d7c7ef7fa7b62d3fb1afe0dc47f2a8f13303cd58d43c709 not found: ID does not exist"
Jul 28 15:49:11.563385 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563361    2580 scope.go:117] "RemoveContainer" containerID="48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"
Jul 28 15:49:11.563595 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:49:11.563577    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b\": container with ID starting with 48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b not found: ID does not exist" containerID="48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"
Jul 28 15:49:11.563658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563605    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b"} err="failed to get container status \"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b\": rpc error: code = NotFound desc = could not find container \"48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b\": container with ID starting with 48d3b42e5076c8ba635a1bfd27bdb3dadbe3c54d39ee5272a17520c24fc9222b not found: ID does not exist"
Jul 28 15:49:11.563658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563628    2580 scope.go:117] "RemoveContainer" containerID="29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"
Jul 28 15:49:11.563889 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:49:11.563874    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a\": container with ID starting with 29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a not found: ID does not exist" containerID="29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"
Jul 28 15:49:11.563931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563892    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a"} err="failed to get container status \"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a\": rpc error: code = NotFound desc = could not find container \"29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a\": container with ID starting with 29898793d1a139325d9959e42da8595cbefbfb5a3144444420650ae742ebc01a not found: ID does not exist"
Jul 28 15:49:11.563931 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.563904    2580 scope.go:117] "RemoveContainer" containerID="7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"
Jul 28 15:49:11.564111 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:49:11.564098    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf\": container with ID starting with 7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf not found: ID does not exist" containerID="7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"
Jul 28 15:49:11.564186 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.564113    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf"} err="failed to get container status \"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf\": rpc error: code = NotFound desc = could not find container \"7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf\": container with ID starting with 7d4eb48ec64186601209ce16ee429d7b6d9391953223832546d6f0f7c683dedf not found: ID does not exist"
Jul 28 15:49:11.567418 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.567396    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:49:11.571710 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:11.571690    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-5rwvr"]
Jul 28 15:49:12.189851 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:12.189815    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" path="/var/lib/kubelet/pods/2b25c6fd-eff1-4aa7-93a9-0b4afea04b12/volumes"
Jul 28 15:49:12.307552 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:49:12.307515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:03.120824 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:03.120783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerStarted","Data":"049a855e6340e452ab93cf3759595baeb729f3ae758dce367c1c10e31d8e7794"}
Jul 28 15:51:03.121303 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:03.120883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerStarted","Data":"ec215a29c51dadbefb794bec421ee34d4af74a4c88f4cb571aeb41e46e84080b"}
Jul 28 15:51:03.121303 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:03.120920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:03.159553 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:03.159491    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podStartSLOduration=2.681167196 podStartE2EDuration="2m23.15944937s" totalImagesPullingTime="1m51.287527518s" totalInitContainerRuntime="29.190754656s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:51:03.157976214 +0000 UTC m=+1077.526629130" watchObservedRunningTime="2026-07-28 15:51:03.15944937 +0000 UTC m=+1077.528102287"
Jul 28 15:51:04.125380 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:04.125346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:04.126556 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:04.126521    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:51:05.131204 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:05.131158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:51:10.137213 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:10.137173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:10.137773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:10.137743    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 15:51:20.138895 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:20.138862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:27.433725 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:27.433690    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:51:27.434212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:27.433989    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container" containerID="cri-o://ec215a29c51dadbefb794bec421ee34d4af74a4c88f4cb571aeb41e46e84080b" gracePeriod=30
Jul 28 15:51:27.434212 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:27.434056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kube-rbac-proxy" containerID="cri-o://049a855e6340e452ab93cf3759595baeb729f3ae758dce367c1c10e31d8e7794" gracePeriod=30
Jul 28 15:51:27.480930 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:27.480900    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:28.239593 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:28.239559    2580 generic.go:356] "Generic (PLEG): container finished" podID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerID="049a855e6340e452ab93cf3759595baeb729f3ae758dce367c1c10e31d8e7794" exitCode=2
Jul 28 15:51:28.239771 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:28.239632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerDied","Data":"049a855e6340e452ab93cf3759595baeb729f3ae758dce367c1c10e31d8e7794"}
Jul 28 15:51:29.246672 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.246639    2580 generic.go:356] "Generic (PLEG): container finished" podID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerID="ec215a29c51dadbefb794bec421ee34d4af74a4c88f4cb571aeb41e46e84080b" exitCode=0
Jul 28 15:51:29.247045 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.246703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerDied","Data":"ec215a29c51dadbefb794bec421ee34d4af74a4c88f4cb571aeb41e46e84080b"}
Jul 28 15:51:29.419602 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.419573    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:29.491443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location\") pod \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") "
Jul 28 15:51:29.491443 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config\") pod \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") "
Jul 28 15:51:29.491715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjddv\" (UniqueName: \"kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv\") pod \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") "
Jul 28 15:51:29.491715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls\") pod \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") "
Jul 28 15:51:29.491715 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert\") pod \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\" (UID: \"fdd4392a-32f8-471a-85a8-cb934b67a3c2\") "
Jul 28 15:51:29.491891 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.491855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdd4392a-32f8-471a-85a8-cb934b67a3c2" (UID: "fdd4392a-32f8-471a-85a8-cb934b67a3c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:51:29.492050 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.492025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdd4392a-32f8-471a-85a8-cb934b67a3c2" (UID: "fdd4392a-32f8-471a-85a8-cb934b67a3c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:51:29.492163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.492049    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:51:29.493688 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.493667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv" (OuterVolumeSpecName: "kube-api-access-pjddv") pod "fdd4392a-32f8-471a-85a8-cb934b67a3c2" (UID: "fdd4392a-32f8-471a-85a8-cb934b67a3c2"). InnerVolumeSpecName "kube-api-access-pjddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:51:29.494061 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.494043    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdd4392a-32f8-471a-85a8-cb934b67a3c2" (UID: "fdd4392a-32f8-471a-85a8-cb934b67a3c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:51:29.497398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.497349    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd4392a-32f8-471a-85a8-cb934b67a3c2" (UID: "fdd4392a-32f8-471a-85a8-cb934b67a3c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:51:29.593058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.593021    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:51:29.593058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.593052    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjddv\" (UniqueName: \"kubernetes.io/projected/fdd4392a-32f8-471a-85a8-cb934b67a3c2-kube-api-access-pjddv\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:51:29.593058 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.593062    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd4392a-32f8-471a-85a8-cb934b67a3c2-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:51:29.593314 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:29.593072    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd4392a-32f8-471a-85a8-cb934b67a3c2-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:51:30.252842 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.252811    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"
Jul 28 15:51:30.252842 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.252820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27" event={"ID":"fdd4392a-32f8-471a-85a8-cb934b67a3c2","Type":"ContainerDied","Data":"a7684925392464460ef2db49136aceb410b1928b920ba3988b49b1026772b33e"}
Jul 28 15:51:30.253403 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.252887    2580 scope.go:117] "RemoveContainer" containerID="049a855e6340e452ab93cf3759595baeb729f3ae758dce367c1c10e31d8e7794"
Jul 28 15:51:30.261536 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.261519    2580 scope.go:117] "RemoveContainer" containerID="ec215a29c51dadbefb794bec421ee34d4af74a4c88f4cb571aeb41e46e84080b"
Jul 28 15:51:30.269494 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.269296    2580 scope.go:117] "RemoveContainer" containerID="e3adae6ffad911284a1c9b9520df1c450811d3efa77daedde1824cf9227d70a9"
Jul 28 15:51:30.273617 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.273596    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:51:30.278531 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:30.278516    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-ztc27"]
Jul 28 15:51:32.186873 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.186839    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" path="/var/lib/kubelet/pods/fdd4392a-32f8-471a-85a8-cb934b67a3c2/volumes"
Jul 28 15:51:32.526522 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.526441    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:32.589277 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:51:32.589613 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589599    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="storage-initializer"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589616    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="storage-initializer"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589629    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kserve-container"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kserve-container"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589644    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589650    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container"
Jul 28 15:51:32.589659 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589658    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589663    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589673    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589678    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589686    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589699    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589705    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="storage-initializer"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589748    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="storage-initializer"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589761    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589766    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589826    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kube-rbac-proxy"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589834    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kserve-container"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589841    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kserve-container"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589848    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a632ca1-e89c-4462-bba0-69ae4efbe47e" containerName="kserve-container"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589853    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="agent"
Jul 28 15:51:32.589857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589860    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b25c6fd-eff1-4aa7-93a9-0b4afea04b12" containerName="kube-rbac-proxy"
Jul 28 15:51:32.590424 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.589866    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd4392a-32f8-471a-85a8-cb934b67a3c2" containerName="kube-rbac-proxy"
Jul 28 15:51:32.605203 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.605177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:51:32.605347 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.605309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.608072 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.608043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 15:51:32.608246 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.608049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 15:51:32.608340 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.608324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:51:32.608408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.608342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:51:32.608467 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.608419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:51:32.609385 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.609356    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:51:32.609385 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.609366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:51:32.718882 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.718847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.719062 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.718897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qwt\" (UniqueName: \"kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.719062 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.718917    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.719062 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.718958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.719062 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.719013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.772397 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.772360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:51:32.820231 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820231 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qwt\" (UniqueName: \"kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820646 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.820898 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.821003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.820975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.822846 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.822827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.829845 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.829825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qwt\" (UniqueName: \"kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-8nw5h\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:32.918857 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:32.918815    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:33.088451 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:33.088418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:51:33.091985 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:51:33.091960    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41eff81_f4a1_4be0_87d3_1797afe0b5ac.slice/crio-b62a138de05638283fb63a1421d58574f42df52604a173c5b9d25e9df45ecb31 WatchSource:0}: Error finding container b62a138de05638283fb63a1421d58574f42df52604a173c5b9d25e9df45ecb31: Status 404 returned error can't find the container with id b62a138de05638283fb63a1421d58574f42df52604a173c5b9d25e9df45ecb31
Jul 28 15:51:33.268990 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:33.268949    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerStarted","Data":"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"}
Jul 28 15:51:33.269377 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:33.269015    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerStarted","Data":"b62a138de05638283fb63a1421d58574f42df52604a173c5b9d25e9df45ecb31"}
Jul 28 15:51:51.354213 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:51.354105    2580 generic.go:356] "Generic (PLEG): container finished" podID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerID="8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0" exitCode=0
Jul 28 15:51:51.354213 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:51.354181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerDied","Data":"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"}
Jul 28 15:51:52.364213 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:52.364176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerStarted","Data":"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"}
Jul 28 15:51:52.364606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:52.364243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerStarted","Data":"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"}
Jul 28 15:51:52.364606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:52.364513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:52.364606 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:52.364585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:51:52.388428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:52.388367    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" podStartSLOduration=2.505058505 podStartE2EDuration="20.388347386s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.883288881s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:51:52.387330575 +0000 UTC m=+1126.755983503" watchObservedRunningTime="2026-07-28 15:51:52.388347386 +0000 UTC m=+1126.757000301"
Jul 28 15:51:58.380534 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:51:58.380500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:52:05.570609 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:52:05.570570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:25.184675 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:25.184644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:53:36.223357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:36.223311    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 15:53:39.090692 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.090655    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:53:39.091096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.090924    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kserve-container" containerID="cri-o://0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323" gracePeriod=30
Jul 28 15:53:39.091096 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.090958    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kube-rbac-proxy" containerID="cri-o://b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb" gracePeriod=30
Jul 28 15:53:39.152285 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.152248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:39.860530 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.860489    2580 generic.go:356] "Generic (PLEG): container finished" podID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerID="b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb" exitCode=2
Jul 28 15:53:39.860716 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:39.860567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerDied","Data":"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"}
Jul 28 15:53:40.538238 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.538213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:53:40.637436 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637342    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5qwt\" (UniqueName: \"kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt\") pod \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") "
Jul 28 15:53:40.637615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location\") pod \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") "
Jul 28 15:53:40.637615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert\") pod \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") "
Jul 28 15:53:40.637615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls\") pod \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") "
Jul 28 15:53:40.637615 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config\") pod \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\" (UID: \"d41eff81-f4a1-4be0-87d3-1797afe0b5ac\") "
Jul 28 15:53:40.637947 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d41eff81-f4a1-4be0-87d3-1797afe0b5ac" (UID: "d41eff81-f4a1-4be0-87d3-1797afe0b5ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:53:40.638033 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.637956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d41eff81-f4a1-4be0-87d3-1797afe0b5ac" (UID: "d41eff81-f4a1-4be0-87d3-1797afe0b5ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:53:40.639783 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.639760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt" (OuterVolumeSpecName: "kube-api-access-z5qwt") pod "d41eff81-f4a1-4be0-87d3-1797afe0b5ac" (UID: "d41eff81-f4a1-4be0-87d3-1797afe0b5ac"). InnerVolumeSpecName "kube-api-access-z5qwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:53:40.639933 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.639920    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d41eff81-f4a1-4be0-87d3-1797afe0b5ac" (UID: "d41eff81-f4a1-4be0-87d3-1797afe0b5ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:53:40.642197 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.642176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d41eff81-f4a1-4be0-87d3-1797afe0b5ac" (UID: "d41eff81-f4a1-4be0-87d3-1797afe0b5ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:53:40.738544 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.738508    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:53:40.738544 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.738543    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:53:40.738544 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.738553    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:53:40.738773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.738561    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:53:40.738773 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.738569    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qwt\" (UniqueName: \"kubernetes.io/projected/d41eff81-f4a1-4be0-87d3-1797afe0b5ac-kube-api-access-z5qwt\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:53:40.866362 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.866325    2580 generic.go:356] "Generic (PLEG): container finished" podID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerID="0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323" exitCode=0
Jul 28 15:53:40.866551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.866399    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"
Jul 28 15:53:40.866551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.866388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerDied","Data":"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"}
Jul 28 15:53:40.866551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.866467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h" event={"ID":"d41eff81-f4a1-4be0-87d3-1797afe0b5ac","Type":"ContainerDied","Data":"b62a138de05638283fb63a1421d58574f42df52604a173c5b9d25e9df45ecb31"}
Jul 28 15:53:40.866551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.866510    2580 scope.go:117] "RemoveContainer" containerID="b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"
Jul 28 15:53:40.875445 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.875424    2580 scope.go:117] "RemoveContainer" containerID="0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"
Jul 28 15:53:40.883029 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.883011    2580 scope.go:117] "RemoveContainer" containerID="8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"
Jul 28 15:53:40.888989 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.888936    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:53:40.891356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.891325    2580 scope.go:117] "RemoveContainer" containerID="b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"
Jul 28 15:53:40.891768 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:53:40.891739    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb\": container with ID starting with b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb not found: ID does not exist" containerID="b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"
Jul 28 15:53:40.891877 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.891778    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb"} err="failed to get container status \"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb\": rpc error: code = NotFound desc = could not find container \"b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb\": container with ID starting with b36e11caef228fb15fa44548cfcd9069ec90b592efb9bf58abebb46c53cbefcb not found: ID does not exist"
Jul 28 15:53:40.891877 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.891804    2580 scope.go:117] "RemoveContainer" containerID="0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"
Jul 28 15:53:40.892147 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:53:40.892107    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323\": container with ID starting with 0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323 not found: ID does not exist" containerID="0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"
Jul 28 15:53:40.892210 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.892157    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323"} err="failed to get container status \"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323\": rpc error: code = NotFound desc = could not find container \"0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323\": container with ID starting with 0b11da88db2b3f8aa9d36a4a5460ed9086cb479392e249281bdbb64a642d7323 not found: ID does not exist"
Jul 28 15:53:40.892210 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.892176    2580 scope.go:117] "RemoveContainer" containerID="8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"
Jul 28 15:53:40.892447 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:53:40.892430    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0\": container with ID starting with 8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0 not found: ID does not exist" containerID="8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"
Jul 28 15:53:40.892506 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.892452    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0"} err="failed to get container status \"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0\": rpc error: code = NotFound desc = could not find container \"8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0\": container with ID starting with 8222ec9d37f496a4bba6689c7aceedbed5efc4ef61071afea78e19ee88a0faa0 not found: ID does not exist"
Jul 28 15:53:40.892981 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:40.892963    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-8nw5h"]
Jul 28 15:53:42.187070 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:42.187035    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" path="/var/lib/kubelet/pods/d41eff81-f4a1-4be0-87d3-1797afe0b5ac/volumes"
Jul 28 15:53:44.161190 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.161158    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:44.222035 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.221992    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:53:44.222551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222531    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kube-rbac-proxy"
Jul 28 15:53:44.222601 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222557    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kube-rbac-proxy"
Jul 28 15:53:44.222636 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222598    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="storage-initializer"
Jul 28 15:53:44.222636 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222608    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="storage-initializer"
Jul 28 15:53:44.222697 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222637    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kserve-container"
Jul 28 15:53:44.222697 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222648    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kserve-container"
Jul 28 15:53:44.222759 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222740    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kube-rbac-proxy"
Jul 28 15:53:44.222759 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.222751    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d41eff81-f4a1-4be0-87d3-1797afe0b5ac" containerName="kserve-container"
Jul 28 15:53:44.227066 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.227041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.229845 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.229818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 15:53:44.231270 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231245    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:53:44.231398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:53:44.231398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:53:44.231398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:53:44.231398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 15:53:44.231398 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.231250    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:53:44.236598 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.236576    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:53:44.371744 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.371711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.371940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.371770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkgj\" (UniqueName: \"kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.371940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.371857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.371940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.371916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.372081 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.371963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkgj\" (UniqueName: \"kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473712 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473712 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.473921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.473894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.474191 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.474165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.474191 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.474172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.476079 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.476058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.484813 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.484786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkgj\" (UniqueName: \"kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.538279 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.538230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:44.678996 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.678917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:53:44.681792 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:53:44.681752    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44427a4_38a4_4b69_843c_b47ebb891d87.slice/crio-8997085a09e7cc3cadfc20052f03fe6f8ca5476a5c37d281901ec4f8988e1beb WatchSource:0}: Error finding container 8997085a09e7cc3cadfc20052f03fe6f8ca5476a5c37d281901ec4f8988e1beb: Status 404 returned error can't find the container with id 8997085a09e7cc3cadfc20052f03fe6f8ca5476a5c37d281901ec4f8988e1beb
Jul 28 15:53:44.887902 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.887865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerStarted","Data":"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"}
Jul 28 15:53:44.888097 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:44.887923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerStarted","Data":"8997085a09e7cc3cadfc20052f03fe6f8ca5476a5c37d281901ec4f8988e1beb"}
Jul 28 15:53:45.449818 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:45.449783    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:53:49.911495 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:49.911401    2580 generic.go:356] "Generic (PLEG): container finished" podID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerID="5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2" exitCode=0
Jul 28 15:53:49.911495 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:49.911475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerDied","Data":"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"}
Jul 28 15:53:50.920517 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:50.920479    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerStarted","Data":"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"}
Jul 28 15:53:50.920949 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:50.920553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerStarted","Data":"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"}
Jul 28 15:53:50.920949 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:50.920765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:50.943764 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:50.943707    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" podStartSLOduration=2.104521395 podStartE2EDuration="6.943693111s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.839171716s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:53:50.941502149 +0000 UTC m=+1245.310155079" watchObservedRunningTime="2026-07-28 15:53:50.943693111 +0000 UTC m=+1245.312346027"
Jul 28 15:53:51.928346 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:51.928313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:53:57.938459 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:53:57.938377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:54:11.437240 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:54:11.437200    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:26.187711 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:26.187628    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:55:37.225322 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:37.225270    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 15:55:41.519948 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:41.519904    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:55:41.520485 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:41.520238    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kserve-container" containerID="cri-o://2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310" gracePeriod=30
Jul 28 15:55:41.520485 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:41.520268    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kube-rbac-proxy" containerID="cri-o://4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210" gracePeriod=30
Jul 28 15:55:41.694453 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:41.694419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:42.415509 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:42.415478    2580 generic.go:356] "Generic (PLEG): container finished" podID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerID="4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210" exitCode=2
Jul 28 15:55:42.415680 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:42.415547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerDied","Data":"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"}
Jul 28 15:55:42.933887 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:42.933837    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 15:55:43.061551 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.061529    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:55:43.124478 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls\") pod \"d44427a4-38a4-4b69-843c-b47ebb891d87\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") "
Jul 28 15:55:43.124478 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config\") pod \"d44427a4-38a4-4b69-843c-b47ebb891d87\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") "
Jul 28 15:55:43.124721 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124539    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert\") pod \"d44427a4-38a4-4b69-843c-b47ebb891d87\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") "
Jul 28 15:55:43.124721 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124637    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location\") pod \"d44427a4-38a4-4b69-843c-b47ebb891d87\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") "
Jul 28 15:55:43.124721 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhkgj\" (UniqueName: \"kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj\") pod \"d44427a4-38a4-4b69-843c-b47ebb891d87\" (UID: \"d44427a4-38a4-4b69-843c-b47ebb891d87\") "
Jul 28 15:55:43.124877 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124821    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d44427a4-38a4-4b69-843c-b47ebb891d87" (UID: "d44427a4-38a4-4b69-843c-b47ebb891d87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:43.124985 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.124958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d44427a4-38a4-4b69-843c-b47ebb891d87" (UID: "d44427a4-38a4-4b69-843c-b47ebb891d87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:55:43.125087 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.125067    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:55:43.125087 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.125086    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d44427a4-38a4-4b69-843c-b47ebb891d87-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:55:43.126751 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.126729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d44427a4-38a4-4b69-843c-b47ebb891d87" (UID: "d44427a4-38a4-4b69-843c-b47ebb891d87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:55:43.127036 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.127017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj" (OuterVolumeSpecName: "kube-api-access-xhkgj") pod "d44427a4-38a4-4b69-843c-b47ebb891d87" (UID: "d44427a4-38a4-4b69-843c-b47ebb891d87"). InnerVolumeSpecName "kube-api-access-xhkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:55:43.129329 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.129305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d44427a4-38a4-4b69-843c-b47ebb891d87" (UID: "d44427a4-38a4-4b69-843c-b47ebb891d87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:55:43.226511 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.226474    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d44427a4-38a4-4b69-843c-b47ebb891d87-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:55:43.226511 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.226507    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhkgj\" (UniqueName: \"kubernetes.io/projected/d44427a4-38a4-4b69-843c-b47ebb891d87-kube-api-access-xhkgj\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:55:43.226706 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.226520    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d44427a4-38a4-4b69-843c-b47ebb891d87-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:55:43.421637 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.421548    2580 generic.go:356] "Generic (PLEG): container finished" podID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerID="2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310" exitCode=0
Jul 28 15:55:43.421637 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.421605    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"
Jul 28 15:55:43.421826 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.421637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerDied","Data":"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"}
Jul 28 15:55:43.421826 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.421706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd" event={"ID":"d44427a4-38a4-4b69-843c-b47ebb891d87","Type":"ContainerDied","Data":"8997085a09e7cc3cadfc20052f03fe6f8ca5476a5c37d281901ec4f8988e1beb"}
Jul 28 15:55:43.421826 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.421735    2580 scope.go:117] "RemoveContainer" containerID="4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"
Jul 28 15:55:43.431029 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.431014    2580 scope.go:117] "RemoveContainer" containerID="2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"
Jul 28 15:55:43.439305 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.439289    2580 scope.go:117] "RemoveContainer" containerID="5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"
Jul 28 15:55:43.447213 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.447192    2580 scope.go:117] "RemoveContainer" containerID="4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"
Jul 28 15:55:43.447496 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:55:43.447478    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210\": container with ID starting with 4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210 not found: ID does not exist" containerID="4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"
Jul 28 15:55:43.447546 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.447505    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210"} err="failed to get container status \"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210\": rpc error: code = NotFound desc = could not find container \"4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210\": container with ID starting with 4c8d05dd5852402b3a8de4fe942724da8e2d9092bf84ce3d3b3bfe01a3f96210 not found: ID does not exist"
Jul 28 15:55:43.447546 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.447524    2580 scope.go:117] "RemoveContainer" containerID="2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"
Jul 28 15:55:43.447831 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:55:43.447806    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310\": container with ID starting with 2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310 not found: ID does not exist" containerID="2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"
Jul 28 15:55:43.447906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.447838    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310"} err="failed to get container status \"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310\": rpc error: code = NotFound desc = could not find container \"2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310\": container with ID starting with 2460aff72cbf10441db3c82e874c82b1731e805771d1f1bea52697d795a04310 not found: ID does not exist"
Jul 28 15:55:43.447906 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.447856    2580 scope.go:117] "RemoveContainer" containerID="5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"
Jul 28 15:55:43.448093 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:55:43.448079    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2\": container with ID starting with 5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2 not found: ID does not exist" containerID="5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"
Jul 28 15:55:43.448161 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.448100    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2"} err="failed to get container status \"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2\": rpc error: code = NotFound desc = could not find container \"5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2\": container with ID starting with 5e15e50080daa42e834d2adaf7f619c0f87d1b44ebf2f0c0eaf658585cab85b2 not found: ID does not exist"
Jul 28 15:55:43.448864 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.448845    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:55:43.460220 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:43.460187    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kgwhd"]
Jul 28 15:55:44.186261 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:44.186222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" path="/var/lib/kubelet/pods/d44427a4-38a4-4b69-843c-b47ebb891d87/volumes"
Jul 28 15:55:46.549337 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.549309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:46.609978 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.609945    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:55:46.610326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610312    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kserve-container"
Jul 28 15:55:46.610376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610327    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kserve-container"
Jul 28 15:55:46.610376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610336    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kube-rbac-proxy"
Jul 28 15:55:46.610376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kube-rbac-proxy"
Jul 28 15:55:46.610376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="storage-initializer"
Jul 28 15:55:46.610376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610358    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="storage-initializer"
Jul 28 15:55:46.610528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610414    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kube-rbac-proxy"
Jul 28 15:55:46.610528 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.610425    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44427a4-38a4-4b69-843c-b47ebb891d87" containerName="kserve-container"
Jul 28 15:55:46.613795 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.613777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.616465 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:55:46.616607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616485    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 15:55:46.616607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:55:46.616741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616721    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:55:46.616741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 15:55:46.616895 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.616869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:55:46.617783 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.617770    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:55:46.627514 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.627492    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:55:46.654321 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.654287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.654502 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.654335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.654502 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.654420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.654502 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.654468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8r9\" (UniqueName: \"kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.654694 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.654526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755490 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8r9\" (UniqueName: \"kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755662 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.755662 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:55:46.755615    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 28 15:55:46.755879 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:55:46.755691    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls podName:114eaf9f-0773-4efb-8d04-fcb2d2198ee1 nodeName:}" failed. No retries permitted until 2026-07-28 15:55:47.255672906 +0000 UTC m=+1361.624325808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 28 15:55:46.755963 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.755944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.756326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.756310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.756376 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.756314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.767118 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.767082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8r9\" (UniqueName: \"kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:46.875402 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:46.875370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:55:47.259844 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:47.259728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:47.262401 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:47.262374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:47.524194 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:47.524078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:55:47.666502 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:47.666469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:55:47.669682 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:55:47.669657    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114eaf9f_0773_4efb_8d04_fcb2d2198ee1.slice/crio-212d414d52601f12a80dbf3f4cc4532a6f28a71be5032f1d239e0622d3bf02da WatchSource:0}: Error finding container 212d414d52601f12a80dbf3f4cc4532a6f28a71be5032f1d239e0622d3bf02da: Status 404 returned error can't find the container with id 212d414d52601f12a80dbf3f4cc4532a6f28a71be5032f1d239e0622d3bf02da
Jul 28 15:55:48.445409 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:48.445374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerStarted","Data":"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"}
Jul 28 15:55:48.445622 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:55:48.445444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerStarted","Data":"212d414d52601f12a80dbf3f4cc4532a6f28a71be5032f1d239e0622d3bf02da"}
Jul 28 15:56:17.592776 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:17.592736    2580 generic.go:356] "Generic (PLEG): container finished" podID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerID="41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a" exitCode=0
Jul 28 15:56:17.593223 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:17.592810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerDied","Data":"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"}
Jul 28 15:56:18.600261 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:18.600226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerStarted","Data":"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"}
Jul 28 15:56:18.600697 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:18.600287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerStarted","Data":"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"}
Jul 28 15:56:18.600697 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:18.600477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:56:19.605194 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:19.605159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:56:25.614232 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:25.614201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:56:25.641676 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:56:25.641609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" podStartSLOduration=10.481069456 podStartE2EDuration="39.641586639s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.160517183s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:56:18.624928163 +0000 UTC m=+1392.993581080" watchObservedRunningTime="2026-07-28 15:56:25.641586639 +0000 UTC m=+1400.010239556"
Jul 28 15:57:56.187176 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:57:56.187144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:58:03.117297 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:03.117264    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:58:03.117796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:03.117480    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kserve-container" containerID="cri-o://eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359" gracePeriod=30
Jul 28 15:58:03.117796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:03.117553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kube-rbac-proxy" containerID="cri-o://47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1" gracePeriod=30
Jul 28 15:58:04.068100 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:04.068066    2580 generic.go:356] "Generic (PLEG): container finished" podID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerID="47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1" exitCode=2
Jul 28 15:58:04.068287 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:04.068157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerDied","Data":"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"}
Jul 28 15:58:04.108808 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:04.108768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:05.609575 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:05.609523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 15:58:06.183336 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:06.183298    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.33:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 15:58:07.663326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.663301    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:58:07.800513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw8r9\" (UniqueName: \"kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9\") pod \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") "
Jul 28 15:58:07.800513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") pod \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") "
Jul 28 15:58:07.800513 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config\") pod \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") "
Jul 28 15:58:07.800796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location\") pod \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") "
Jul 28 15:58:07.800796 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert\") pod \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\" (UID: \"114eaf9f-0773-4efb-8d04-fcb2d2198ee1\") "
Jul 28 15:58:07.800956 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "114eaf9f-0773-4efb-8d04-fcb2d2198ee1" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:07.801016 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.800973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "114eaf9f-0773-4efb-8d04-fcb2d2198ee1" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:07.802686 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.802662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "114eaf9f-0773-4efb-8d04-fcb2d2198ee1" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:58:07.802905 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.802880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9" (OuterVolumeSpecName: "kube-api-access-tw8r9") pod "114eaf9f-0773-4efb-8d04-fcb2d2198ee1" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1"). InnerVolumeSpecName "kube-api-access-tw8r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:58:07.805040 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.805017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "114eaf9f-0773-4efb-8d04-fcb2d2198ee1" (UID: "114eaf9f-0773-4efb-8d04-fcb2d2198ee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:58:07.901253 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.901213    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8r9\" (UniqueName: \"kubernetes.io/projected/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-api-access-tw8r9\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:07.901253 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.901243    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:07.901253 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.901257    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:07.901485 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.901266    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:07.901485 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:07.901276    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/114eaf9f-0773-4efb-8d04-fcb2d2198ee1-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:08.088159 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.088110    2580 generic.go:356] "Generic (PLEG): container finished" podID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerID="eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359" exitCode=0
Jul 28 15:58:08.088348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.088197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"
Jul 28 15:58:08.088348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.088196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerDied","Data":"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"}
Jul 28 15:58:08.088348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.088269    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4" event={"ID":"114eaf9f-0773-4efb-8d04-fcb2d2198ee1","Type":"ContainerDied","Data":"212d414d52601f12a80dbf3f4cc4532a6f28a71be5032f1d239e0622d3bf02da"}
Jul 28 15:58:08.088348 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.088311    2580 scope.go:117] "RemoveContainer" containerID="47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"
Jul 28 15:58:08.097772 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.097757    2580 scope.go:117] "RemoveContainer" containerID="eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"
Jul 28 15:58:08.105636 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.105614    2580 scope.go:117] "RemoveContainer" containerID="41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"
Jul 28 15:58:08.112787 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.112752    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:58:08.115597 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.115540    2580 scope.go:117] "RemoveContainer" containerID="47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"
Jul 28 15:58:08.116369 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:08.116215    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1\": container with ID starting with 47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1 not found: ID does not exist" containerID="47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"
Jul 28 15:58:08.116369 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.116260    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1"} err="failed to get container status \"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1\": rpc error: code = NotFound desc = could not find container \"47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1\": container with ID starting with 47f4f1b66890b3f779799290ca3dcdee91f58b9cfc8a28bdf41b64ac544c47b1 not found: ID does not exist"
Jul 28 15:58:08.116369 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.116291    2580 scope.go:117] "RemoveContainer" containerID="eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"
Jul 28 15:58:08.116658 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:08.116589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359\": container with ID starting with eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359 not found: ID does not exist" containerID="eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"
Jul 28 15:58:08.116658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.116612    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359"} err="failed to get container status \"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359\": rpc error: code = NotFound desc = could not find container \"eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359\": container with ID starting with eade127ae669bbab8ba7d25f092ea7187c26801e03abc403a40f816bd7e87359 not found: ID does not exist"
Jul 28 15:58:08.116658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.116629    2580 scope.go:117] "RemoveContainer" containerID="41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"
Jul 28 15:58:08.116923 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:08.116904    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a\": container with ID starting with 41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a not found: ID does not exist" containerID="41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"
Jul 28 15:58:08.116995 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.116929    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a"} err="failed to get container status \"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a\": rpc error: code = NotFound desc = could not find container \"41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a\": container with ID starting with 41d72eea9606763d66b8291d8e6be1ee8e4ba8e2711e83d60c3f92efe71d706a not found: ID does not exist"
Jul 28 15:58:08.117964 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.117947    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-rkpt4"]
Jul 28 15:58:08.186958 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.186929    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" path="/var/lib/kubelet/pods/114eaf9f-0773-4efb-8d04-fcb2d2198ee1/volumes"
Jul 28 15:58:08.278868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.278835    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:08.354356 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354277    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:08.354619 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="storage-initializer"
Jul 28 15:58:08.354666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354621    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="storage-initializer"
Jul 28 15:58:08.354666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354638    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kube-rbac-proxy"
Jul 28 15:58:08.354666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354644    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kube-rbac-proxy"
Jul 28 15:58:08.354666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354659    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kserve-container"
Jul 28 15:58:08.354666 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354666    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kserve-container"
Jul 28 15:58:08.354809 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kserve-container"
Jul 28 15:58:08.354809 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.354723    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="114eaf9f-0773-4efb-8d04-fcb2d2198ee1" containerName="kube-rbac-proxy"
Jul 28 15:58:08.359747 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.359731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363198    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363340    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363461    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.363900    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:58:08.366408 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.364050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 15:58:08.369785 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.369761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:08.506622 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.506589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.506799 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.506655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.506799 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.506692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.506799 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.506716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmwm\" (UniqueName: \"kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.506799 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.506745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.553248 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.553215    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:08.608011 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.607936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608011 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.607973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmwm\" (UniqueName: \"kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608011 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.607999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608301 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.608028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608301 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.608103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608446 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.608420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608717 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.608691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.608821 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.608752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.610717 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.610702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.618634 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.618614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmwm\" (UniqueName: \"kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.676934 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.676901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:08.812782 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:08.812759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:08.814710 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:58:08.814675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239e0bc9_00f4_4717_b6f3_6fc2d7d993fd.slice/crio-0daaf67f3699bfd508364cc9d2ae971745d19ce4658f5654d64a8b8ed09d907a WatchSource:0}: Error finding container 0daaf67f3699bfd508364cc9d2ae971745d19ce4658f5654d64a8b8ed09d907a: Status 404 returned error can't find the container with id 0daaf67f3699bfd508364cc9d2ae971745d19ce4658f5654d64a8b8ed09d907a
Jul 28 15:58:09.096520 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:09.096483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerStarted","Data":"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"}
Jul 28 15:58:09.096681 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:09.096543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerStarted","Data":"0daaf67f3699bfd508364cc9d2ae971745d19ce4658f5654d64a8b8ed09d907a"}
Jul 28 15:58:18.140925 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:18.140889    2580 generic.go:356] "Generic (PLEG): container finished" podID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerID="a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5" exitCode=0
Jul 28 15:58:18.141389 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:18.140937    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerDied","Data":"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"}
Jul 28 15:58:19.149017 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:19.148971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerStarted","Data":"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"}
Jul 28 15:58:19.149407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:19.149059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerStarted","Data":"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"}
Jul 28 15:58:19.149407 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:19.149292    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:19.171922 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:19.171859    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podStartSLOduration=2.354504032 podStartE2EDuration="11.1718362s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.817332168s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:58:19.169956333 +0000 UTC m=+1513.538609250" watchObservedRunningTime="2026-07-28 15:58:19.1718362 +0000 UTC m=+1513.540489117"
Jul 28 15:58:20.155107 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:20.155071    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:20.156335 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:20.156306    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 15:58:21.160084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:21.160045    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 15:58:26.164714 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:26.164630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:26.167059 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:26.165264    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 15:58:36.166320 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:36.166288    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:39.182833 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:39.182799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:50.491006 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:50.490968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:50.916383 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:50.916348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:50.916603 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:50.916570    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container" containerID="cri-o://7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3" gracePeriod=30
Jul 28 15:58:50.916736 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:50.916665    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kube-rbac-proxy" containerID="cri-o://24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d" gracePeriod=30
Jul 28 15:58:51.160344 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:51.160302    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 15:58:51.308240 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:51.308156    2580 generic.go:356] "Generic (PLEG): container finished" podID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerID="24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d" exitCode=2
Jul 28 15:58:51.308392 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:51.308231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerDied","Data":"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"}
Jul 28 15:58:51.954361 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:51.954311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:52.664634 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.664608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:52.796912 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.796827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls\") pod \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") "
Jul 28 15:58:52.796912 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.796863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert\") pod \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") "
Jul 28 15:58:52.797106 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.796915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjmwm\" (UniqueName: \"kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm\") pod \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") "
Jul 28 15:58:52.797106 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.796969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config\") pod \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") "
Jul 28 15:58:52.797220 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.797178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location\") pod \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\" (UID: \"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd\") "
Jul 28 15:58:52.797374 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.797337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" (UID: "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:52.797512 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.797344    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" (UID: "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:58:52.798081 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.798054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" (UID: "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:58:52.799270 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.799250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm" (OuterVolumeSpecName: "kube-api-access-cjmwm") pod "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" (UID: "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd"). InnerVolumeSpecName "kube-api-access-cjmwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:58:52.799270 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.799253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" (UID: "239e0bc9-00f4-4717-b6f3-6fc2d7d993fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:58:52.898479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.898440    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmwm\" (UniqueName: \"kubernetes.io/projected/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-api-access-cjmwm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:52.898479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.898474    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:52.898479 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.898486    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:52.898730 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.898498    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:52.898730 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:52.898511    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:58:53.320858 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.320822    2580 generic.go:356] "Generic (PLEG): container finished" podID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerID="7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3" exitCode=0
Jul 28 15:58:53.321357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.320866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerDied","Data":"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"}
Jul 28 15:58:53.321357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.320884    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"
Jul 28 15:58:53.321357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.320916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d" event={"ID":"239e0bc9-00f4-4717-b6f3-6fc2d7d993fd","Type":"ContainerDied","Data":"0daaf67f3699bfd508364cc9d2ae971745d19ce4658f5654d64a8b8ed09d907a"}
Jul 28 15:58:53.321357 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.320941    2580 scope.go:117] "RemoveContainer" containerID="24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"
Jul 28 15:58:53.330048 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.330031    2580 scope.go:117] "RemoveContainer" containerID="7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"
Jul 28 15:58:53.337757 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.337739    2580 scope.go:117] "RemoveContainer" containerID="a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"
Jul 28 15:58:53.344275 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.344252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:53.345867 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.345851    2580 scope.go:117] "RemoveContainer" containerID="24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"
Jul 28 15:58:53.346202 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:53.346179    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d\": container with ID starting with 24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d not found: ID does not exist" containerID="24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"
Jul 28 15:58:53.346261 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.346215    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d"} err="failed to get container status \"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d\": rpc error: code = NotFound desc = could not find container \"24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d\": container with ID starting with 24d9bca9c7780c644fc2b9cf4e036389673a0e6c2019f241b77a45e1f0fa2c7d not found: ID does not exist"
Jul 28 15:58:53.346261 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.346244    2580 scope.go:117] "RemoveContainer" containerID="7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"
Jul 28 15:58:53.346504 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:53.346486    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3\": container with ID starting with 7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3 not found: ID does not exist" containerID="7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"
Jul 28 15:58:53.346572 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.346512    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3"} err="failed to get container status \"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3\": rpc error: code = NotFound desc = could not find container \"7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3\": container with ID starting with 7134d9ab8c27e3b8d6377caf802941cb714eb719649a16a97ebf4e47a57f25d3 not found: ID does not exist"
Jul 28 15:58:53.346572 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.346534    2580 scope.go:117] "RemoveContainer" containerID="a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"
Jul 28 15:58:53.346836 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:53.346818    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5\": container with ID starting with a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5 not found: ID does not exist" containerID="a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"
Jul 28 15:58:53.346885 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.346840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5"} err="failed to get container status \"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5\": rpc error: code = NotFound desc = could not find container \"a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5\": container with ID starting with a1196024546cd29c76cd9ce59d090fb47022860f636640b0b66e429bdcdc6ec5 not found: ID does not exist"
Jul 28 15:58:53.349547 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:53.349528    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-cxz8d"]
Jul 28 15:58:54.190076 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:54.190031    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" path="/var/lib/kubelet/pods/239e0bc9-00f4-4717-b6f3-6fc2d7d993fd/volumes"
Jul 28 15:58:56.019590 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.019554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:56.069587 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069550    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:58:56.069913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069901    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container"
Jul 28 15:58:56.069960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069915    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container"
Jul 28 15:58:56.069960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069931    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kube-rbac-proxy"
Jul 28 15:58:56.069960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069936    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kube-rbac-proxy"
Jul 28 15:58:56.069960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069944    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="storage-initializer"
Jul 28 15:58:56.069960 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069949    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="storage-initializer"
Jul 28 15:58:56.070102 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.069997    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kserve-container"
Jul 28 15:58:56.070102 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.070007    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="239e0bc9-00f4-4717-b6f3-6fc2d7d993fd" containerName="kube-rbac-proxy"
Jul 28 15:58:56.073292 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.073273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.075921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.075895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:58:56.075921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.075917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:58:56.076110 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.075941    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 15:58:56.076110 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.075917    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:58:56.076248 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.076231    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 15:58:56.076248 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.076241    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:58:56.077112 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.077097    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:58:56.084053 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.084027    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:58:56.131938 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.131894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.131938 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.131937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrd9f\" (UniqueName: \"kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.132209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.131979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.132209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.132022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.132209 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.132041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232589 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.232556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232589 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.232591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrd9f\" (UniqueName: \"kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.232627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.232671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.232695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.232854 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:56.232810    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 15:58:56.233048 ip-10-0-140-97 kubenswrapper[2580]: E0728 15:58:56.232896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls podName:0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e nodeName:}" failed. No retries permitted until 2026-07-28 15:58:56.732873843 +0000 UTC m=+1551.101526737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 15:58:56.233110 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.233092    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.233370 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.233353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.233414 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.233356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.242148 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.242099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrd9f\" (UniqueName: \"kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.605471 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.605435    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:58:56.736883 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.736848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.739527 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.739506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:56.983962 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:56.983877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:58:57.121098 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:57.121072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:58:57.122714 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:58:57.122686    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf5dea7_5d37_4d0b_ba22_6d10783f4d5e.slice/crio-81c32ccc514c32eb3c3307bd0ee406a22f6dd8f411b0bbe11e1353f115ac9550 WatchSource:0}: Error finding container 81c32ccc514c32eb3c3307bd0ee406a22f6dd8f411b0bbe11e1353f115ac9550: Status 404 returned error can't find the container with id 81c32ccc514c32eb3c3307bd0ee406a22f6dd8f411b0bbe11e1353f115ac9550
Jul 28 15:58:57.345041 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:57.345005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerStarted","Data":"a7b88e2d7088f0292e5b470c0080111bbff2d97c71f2874ff730cf796fac4a21"}
Jul 28 15:58:57.345235 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:58:57.345065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerStarted","Data":"81c32ccc514c32eb3c3307bd0ee406a22f6dd8f411b0bbe11e1353f115ac9550"}
Jul 28 15:59:02.369779 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:02.369748    2580 generic.go:356] "Generic (PLEG): container finished" podID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerID="a7b88e2d7088f0292e5b470c0080111bbff2d97c71f2874ff730cf796fac4a21" exitCode=0
Jul 28 15:59:02.370174 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:02.369830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerDied","Data":"a7b88e2d7088f0292e5b470c0080111bbff2d97c71f2874ff730cf796fac4a21"}
Jul 28 15:59:03.377455 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.377418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerStarted","Data":"0c0a0a5cb9c6da50af87d575f04586e9e44a29991de3619800f5e9d06e3f6ee3"}
Jul 28 15:59:03.377868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.377495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerStarted","Data":"47c917617155a7dccb166cc4eca10d096e60dfefb9a207ee96c586894e7f9fff"}
Jul 28 15:59:03.377868 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.377840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:03.377999 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.377978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:03.379596 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.379574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 15:59:03.409014 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:03.408968    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podStartSLOduration=3.009525939 podStartE2EDuration="7.408951176s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.399425237s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:59:03.407428191 +0000 UTC m=+1557.776081108" watchObservedRunningTime="2026-07-28 15:59:03.408951176 +0000 UTC m=+1557.777604092"
Jul 28 15:59:04.382421 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:04.382375    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 15:59:09.387084 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:09.387053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:09.387639 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:09.387608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 15:59:19.388390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:19.388334    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:31.280299 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.280256    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:59:31.280742 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.280544    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kube-rbac-proxy" containerID="cri-o://0c0a0a5cb9c6da50af87d575f04586e9e44a29991de3619800f5e9d06e3f6ee3" gracePeriod=30
Jul 28 15:59:31.280742 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.280549    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container" containerID="cri-o://47c917617155a7dccb166cc4eca10d096e60dfefb9a207ee96c586894e7f9fff" gracePeriod=30
Jul 28 15:59:31.328849 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.328821    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:31.505193 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.505158    2580 generic.go:356] "Generic (PLEG): container finished" podID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerID="0c0a0a5cb9c6da50af87d575f04586e9e44a29991de3619800f5e9d06e3f6ee3" exitCode=2
Jul 28 15:59:31.505193 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:31.505196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerDied","Data":"0c0a0a5cb9c6da50af87d575f04586e9e44a29991de3619800f5e9d06e3f6ee3"}
Jul 28 15:59:32.512616 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.512572    2580 generic.go:356] "Generic (PLEG): container finished" podID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerID="47c917617155a7dccb166cc4eca10d096e60dfefb9a207ee96c586894e7f9fff" exitCode=0
Jul 28 15:59:32.512963 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.512645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerDied","Data":"47c917617155a7dccb166cc4eca10d096e60dfefb9a207ee96c586894e7f9fff"}
Jul 28 15:59:32.625761 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.625737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:32.762201 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config\") pod \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") "
Jul 28 15:59:32.762201 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrd9f\" (UniqueName: \"kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f\") pod \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") "
Jul 28 15:59:32.762441 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location\") pod \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") "
Jul 28 15:59:32.762441 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762254    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") pod \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") "
Jul 28 15:59:32.762441 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert\") pod \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\" (UID: \"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e\") "
Jul 28 15:59:32.762591 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762508    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:59:32.762641 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.762617    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 15:59:32.763266 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.763244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 15:59:32.764487 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.764468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f" (OuterVolumeSpecName: "kube-api-access-jrd9f") pod "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e"). InnerVolumeSpecName "kube-api-access-jrd9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 15:59:32.764549 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.764511    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" (UID: "0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 15:59:32.863107 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.863063    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:59:32.863107 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.863106    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrd9f\" (UniqueName: \"kubernetes.io/projected/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kube-api-access-jrd9f\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:59:32.863107 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.863115    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:59:32.863351 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.863151    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:59:32.863351 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:32.863162    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 15:59:33.518658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.518629    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"
Jul 28 15:59:33.518658 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.518642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm" event={"ID":"0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e","Type":"ContainerDied","Data":"81c32ccc514c32eb3c3307bd0ee406a22f6dd8f411b0bbe11e1353f115ac9550"}
Jul 28 15:59:33.519108 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.518748    2580 scope.go:117] "RemoveContainer" containerID="0c0a0a5cb9c6da50af87d575f04586e9e44a29991de3619800f5e9d06e3f6ee3"
Jul 28 15:59:33.527677 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.527659    2580 scope.go:117] "RemoveContainer" containerID="47c917617155a7dccb166cc4eca10d096e60dfefb9a207ee96c586894e7f9fff"
Jul 28 15:59:33.535752 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.535726    2580 scope.go:117] "RemoveContainer" containerID="a7b88e2d7088f0292e5b470c0080111bbff2d97c71f2874ff730cf796fac4a21"
Jul 28 15:59:33.544935 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.544910    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:59:33.549929 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:33.549906    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-d4xrm"]
Jul 28 15:59:34.186315 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:34.186283    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" path="/var/lib/kubelet/pods/0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e/volumes"
Jul 28 15:59:36.454332 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.454292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:36.497635 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.497599    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 15:59:36.498003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.497987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container"
Jul 28 15:59:36.498003 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498004    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498028    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="storage-initializer"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498033    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="storage-initializer"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498040    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kube-rbac-proxy"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498048    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kube-rbac-proxy"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498115    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kube-rbac-proxy"
Jul 28 15:59:36.498163 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.498139    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf5dea7-5d37-4d0b-ba22-6d10783f4d5e" containerName="kserve-container"
Jul 28 15:59:36.501269 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.501255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.504463 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.504441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 15:59:36.504567 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.504523    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 15:59:36.504995 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.504973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 15:59:36.505301 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.505286    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 15:59:36.505500 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.505478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 15:59:36.505579 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.505487    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 15:59:36.505639 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.505611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 15:59:36.512782 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.512760    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 15:59:36.542326 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.542238    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:36.544000 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.543976    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 15:59:36.547908 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.547888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.550617 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.550600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 15:59:36.557993 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.557973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 15:59:36.596938 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.596905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.596938 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.596940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz89s\" (UniqueName: \"kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.597168 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.596962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.597168 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.597099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.597168 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.597152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.629139 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.629099    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:36.698501 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27gw\" (UniqueName: \"kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.698681 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.698681 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz89s\" (UniqueName: \"kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.698681 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.698854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.698854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.698854 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.698953 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.698987 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.699021 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.698990    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.699252 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.699237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.699300 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.699273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.699337 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.699313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.701397 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.701380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.707626 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.707567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz89s\" (UniqueName: \"kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s\") pod \"isvc-canary-create-predictor-cb85cf55c-kbtvt\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.725812 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.725785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 15:59:36.800292 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27gw\" (UniqueName: \"kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.800457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.800457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.800457 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.800610 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.800921 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.800899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.801036 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.801018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.801081 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.801040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.802922 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.802903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.809890 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.809869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27gw\" (UniqueName: \"kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-btngv\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.811721 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.811691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:36.859135 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.859081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:36.958115 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:36.958085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 15:59:36.961288 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:59:36.961259    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617cba57_d003_44d8_8c64_ba0e7db44155.slice/crio-bf0c0fb36f1a29b0c3a14cad5cf9e5c5d1de3d161d4d52a26b28a7f2cf296354 WatchSource:0}: Error finding container bf0c0fb36f1a29b0c3a14cad5cf9e5c5d1de3d161d4d52a26b28a7f2cf296354: Status 404 returned error can't find the container with id bf0c0fb36f1a29b0c3a14cad5cf9e5c5d1de3d161d4d52a26b28a7f2cf296354
Jul 28 15:59:37.008281 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:37.008258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 15:59:37.010487 ip-10-0-140-97 kubenswrapper[2580]: W0728 15:59:37.010461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c6860b_d806_4d94_9c84_39777f2f2638.slice/crio-046302b2f7525b752625753ad2e7fc74379c33080672f8078cf89de931e84616 WatchSource:0}: Error finding container 046302b2f7525b752625753ad2e7fc74379c33080672f8078cf89de931e84616: Status 404 returned error can't find the container with id 046302b2f7525b752625753ad2e7fc74379c33080672f8078cf89de931e84616
Jul 28 15:59:37.545913 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:37.545876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerStarted","Data":"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"}
Jul 28 15:59:37.546411 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:37.545943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerStarted","Data":"bf0c0fb36f1a29b0c3a14cad5cf9e5c5d1de3d161d4d52a26b28a7f2cf296354"}
Jul 28 15:59:37.547644 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:37.547614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerStarted","Data":"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"}
Jul 28 15:59:37.547741 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:37.547676    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerStarted","Data":"046302b2f7525b752625753ad2e7fc74379c33080672f8078cf89de931e84616"}
Jul 28 15:59:41.567940 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:41.567905    2580 generic.go:356] "Generic (PLEG): container finished" podID="617cba57-d003-44d8-8c64-ba0e7db44155" containerID="b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb" exitCode=0
Jul 28 15:59:41.568469 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:41.567994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerDied","Data":"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"}
Jul 28 15:59:41.569791 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:41.569770    2580 generic.go:356] "Generic (PLEG): container finished" podID="27c6860b-d806-4d94-9c84-39777f2f2638" containerID="f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d" exitCode=0
Jul 28 15:59:41.569874 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:41.569815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerDied","Data":"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"}
Jul 28 15:59:42.576428 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.576395    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerStarted","Data":"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"}
Jul 28 15:59:42.576881 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.576457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerStarted","Data":"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"}
Jul 28 15:59:42.576881 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.576665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:42.578570 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.578544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerStarted","Data":"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"}
Jul 28 15:59:42.578682 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.578592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerStarted","Data":"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"}
Jul 28 15:59:42.578853 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.578837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:42.578918 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.578879    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:42.580229 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.580203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 15:59:42.598392 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.598340    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podStartSLOduration=2.734007015 podStartE2EDuration="6.598321446s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864314431s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:59:42.596392876 +0000 UTC m=+1596.965045816" watchObservedRunningTime="2026-07-28 15:59:42.598321446 +0000 UTC m=+1596.966974365"
Jul 28 15:59:42.617336 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:42.617275    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podStartSLOduration=2.931127899 podStartE2EDuration="6.617259859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68613196s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:59:42.616430503 +0000 UTC m=+1596.985083419" watchObservedRunningTime="2026-07-28 15:59:42.617259859 +0000 UTC m=+1596.985912817"
Jul 28 15:59:43.584182 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:43.584149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:43.584182 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:43.584171    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 15:59:43.585389 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:43.585362    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 15:59:44.588607 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:44.588566    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 15:59:48.588843 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:48.588816    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 15:59:48.589390 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:48.589361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 15:59:49.592930 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:49.592900    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 15:59:49.593447 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:49.593422    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 15:59:58.590294 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:58.590208    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 15:59:59.593448 ip-10-0-140-97 kubenswrapper[2580]: I0728 15:59:59.593406    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:08.589742 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:08.589700    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:00:09.594004 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:09.593957    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:18.589690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:18.589646    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:00:19.594277 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:19.594232    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:21.870861 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:21.870825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:00:21.934437 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:21.934408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:00:28.589978 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:28.589937    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:00:29.593989 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:29.593946    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:38.590060 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:38.590014    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:00:39.593567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:39.593526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:48.589365 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:48.589329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:00:49.593394 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:49.593355    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:00:57.182187 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:57.182123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 16:00:58.590409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:00:58.590375    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 16:01:06.688635 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.688602    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 16:01:06.689027 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.688901    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" containerID="cri-o://77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd" gracePeriod=30
Jul 28 16:01:06.689027 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.688922    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kube-rbac-proxy" containerID="cri-o://ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c" gracePeriod=30
Jul 28 16:01:06.693518 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.693491    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 16:01:06.693751 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.693714    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" containerID="cri-o://9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b" gracePeriod=30
Jul 28 16:01:06.693835 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.693773    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kube-rbac-proxy" containerID="cri-o://e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456" gracePeriod=30
Jul 28 16:01:06.709528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.709502    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:06.791501 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.791465    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:06.979062 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.978972    2580 generic.go:356] "Generic (PLEG): container finished" podID="27c6860b-d806-4d94-9c84-39777f2f2638" containerID="e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456" exitCode=2
Jul 28 16:01:06.979062 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.979044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerDied","Data":"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"}
Jul 28 16:01:06.981195 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.981173    2580 generic.go:356] "Generic (PLEG): container finished" podID="617cba57-d003-44d8-8c64-ba0e7db44155" containerID="ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c" exitCode=2
Jul 28 16:01:06.981303 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:06.981202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerDied","Data":"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"}
Jul 28 16:01:07.181512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:07.181473    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:01:08.585111 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:08.585072    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 16:01:08.589422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:08.589396    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 16:01:09.588674 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:09.588622    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 16:01:11.638099 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.638072    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 16:01:11.742871 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.742762    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:11.754777 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.754754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location\") pod \"617cba57-d003-44d8-8c64-ba0e7db44155\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") "
Jul 28 16:01:11.754896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.754832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config\") pod \"617cba57-d003-44d8-8c64-ba0e7db44155\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") "
Jul 28 16:01:11.754896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.754870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz89s\" (UniqueName: \"kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s\") pod \"617cba57-d003-44d8-8c64-ba0e7db44155\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") "
Jul 28 16:01:11.755021 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.754896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls\") pod \"617cba57-d003-44d8-8c64-ba0e7db44155\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") "
Jul 28 16:01:11.755021 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.754976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert\") pod \"617cba57-d003-44d8-8c64-ba0e7db44155\" (UID: \"617cba57-d003-44d8-8c64-ba0e7db44155\") "
Jul 28 16:01:11.755157 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.755110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "617cba57-d003-44d8-8c64-ba0e7db44155" (UID: "617cba57-d003-44d8-8c64-ba0e7db44155"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:01:11.755292 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.755203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "617cba57-d003-44d8-8c64-ba0e7db44155" (UID: "617cba57-d003-44d8-8c64-ba0e7db44155"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:11.755391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.755291    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/617cba57-d003-44d8-8c64-ba0e7db44155-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.755391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.755309    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.755473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.755446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "617cba57-d003-44d8-8c64-ba0e7db44155" (UID: "617cba57-d003-44d8-8c64-ba0e7db44155"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:11.757109 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.757080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s" (OuterVolumeSpecName: "kube-api-access-pz89s") pod "617cba57-d003-44d8-8c64-ba0e7db44155" (UID: "617cba57-d003-44d8-8c64-ba0e7db44155"). InnerVolumeSpecName "kube-api-access-pz89s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:01:11.757239 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.757161    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "617cba57-d003-44d8-8c64-ba0e7db44155" (UID: "617cba57-d003-44d8-8c64-ba0e7db44155"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:01:11.813028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.812994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:01:11.813404 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813391    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container"
Jul 28 16:01:11.813450 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813406    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container"
Jul 28 16:01:11.813450 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813423    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="storage-initializer"
Jul 28 16:01:11.813450 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813428    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="storage-initializer"
Jul 28 16:01:11.813450 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813435    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kube-rbac-proxy"
Jul 28 16:01:11.813450 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813441    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kube-rbac-proxy"
Jul 28 16:01:11.813598 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813522    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kube-rbac-proxy"
Jul 28 16:01:11.813598 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.813536    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" containerName="kserve-container"
Jul 28 16:01:11.817059 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.817042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.820268 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.820242    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 16:01:11.820501 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.820246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 16:01:11.823285 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.823265    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:01:11.827532 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.827510    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:01:11.827630 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.827601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.830169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.830120    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 16:01:11.841213 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.841189    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:01:11.845849 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.845829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 16:01:11.856087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.856066    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/617cba57-d003-44d8-8c64-ba0e7db44155-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.856203 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.856091    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz89s\" (UniqueName: \"kubernetes.io/projected/617cba57-d003-44d8-8c64-ba0e7db44155-kube-api-access-pz89s\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.856203 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.856105    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/617cba57-d003-44d8-8c64-ba0e7db44155-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.870790 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.870765    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:11.956688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.956647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config\") pod \"27c6860b-d806-4d94-9c84-39777f2f2638\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") "
Jul 28 16:01:11.956883 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.956737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location\") pod \"27c6860b-d806-4d94-9c84-39777f2f2638\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") "
Jul 28 16:01:11.956883 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.956768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q27gw\" (UniqueName: \"kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw\") pod \"27c6860b-d806-4d94-9c84-39777f2f2638\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") "
Jul 28 16:01:11.956883 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.956799    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert\") pod \"27c6860b-d806-4d94-9c84-39777f2f2638\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") "
Jul 28 16:01:11.957052 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.956891    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls\") pod \"27c6860b-d806-4d94-9c84-39777f2f2638\" (UID: \"27c6860b-d806-4d94-9c84-39777f2f2638\") "
Jul 28 16:01:11.957052 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27c6860b-d806-4d94-9c84-39777f2f2638" (UID: "27c6860b-d806-4d94-9c84-39777f2f2638"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:11.957052 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.957233 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27c6860b-d806-4d94-9c84-39777f2f2638" (UID: "27c6860b-d806-4d94-9c84-39777f2f2638"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:01:11.957233 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.957329 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.957329 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27c6860b-d806-4d94-9c84-39777f2f2638" (UID: "27c6860b-d806-4d94-9c84-39777f2f2638"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:01:11.957410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.957410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.957487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.957487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.957487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.957605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbrr\" (UniqueName: \"kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:11.957706 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcsg\" (UniqueName: \"kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:11.957810 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957795    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.957864 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957818    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27c6860b-d806-4d94-9c84-39777f2f2638-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.957864 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.957834    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27c6860b-d806-4d94-9c84-39777f2f2638-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:11.959025 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.959008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27c6860b-d806-4d94-9c84-39777f2f2638" (UID: "27c6860b-d806-4d94-9c84-39777f2f2638"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:01:11.959099 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.959031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw" (OuterVolumeSpecName: "kube-api-access-q27gw") pod "27c6860b-d806-4d94-9c84-39777f2f2638" (UID: "27c6860b-d806-4d94-9c84-39777f2f2638"). InnerVolumeSpecName "kube-api-access-q27gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:01:11.976338 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:11.976311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:12.005051 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.004969    2580 generic.go:356] "Generic (PLEG): container finished" podID="617cba57-d003-44d8-8c64-ba0e7db44155" containerID="77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd" exitCode=0
Jul 28 16:01:12.005051 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.005031    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"
Jul 28 16:01:12.005275 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.005053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerDied","Data":"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"}
Jul 28 16:01:12.005275 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.005111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt" event={"ID":"617cba57-d003-44d8-8c64-ba0e7db44155","Type":"ContainerDied","Data":"bf0c0fb36f1a29b0c3a14cad5cf9e5c5d1de3d161d4d52a26b28a7f2cf296354"}
Jul 28 16:01:12.005275 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.005152    2580 scope.go:117] "RemoveContainer" containerID="ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"
Jul 28 16:01:12.007238 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.007211    2580 generic.go:356] "Generic (PLEG): container finished" podID="27c6860b-d806-4d94-9c84-39777f2f2638" containerID="9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b" exitCode=0
Jul 28 16:01:12.007370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.007243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerDied","Data":"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"}
Jul 28 16:01:12.007370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.007308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv" event={"ID":"27c6860b-d806-4d94-9c84-39777f2f2638","Type":"ContainerDied","Data":"046302b2f7525b752625753ad2e7fc74379c33080672f8078cf89de931e84616"}
Jul 28 16:01:12.007602 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.007577    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"
Jul 28 16:01:12.015251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.015218    2580 scope.go:117] "RemoveContainer" containerID="77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"
Jul 28 16:01:12.023301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.023271    2580 scope.go:117] "RemoveContainer" containerID="b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"
Jul 28 16:01:12.030072 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.030045    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 16:01:12.031541 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.031527    2580 scope.go:117] "RemoveContainer" containerID="ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"
Jul 28 16:01:12.031850 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.031831    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c\": container with ID starting with ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c not found: ID does not exist" containerID="ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"
Jul 28 16:01:12.031900 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.031857    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c"} err="failed to get container status \"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c\": rpc error: code = NotFound desc = could not find container \"ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c\": container with ID starting with ec1c8d81700d919531e8b02cd75b79c011d793ad141a25998d06d681f1fce45c not found: ID does not exist"
Jul 28 16:01:12.031900 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.031874    2580 scope.go:117] "RemoveContainer" containerID="77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"
Jul 28 16:01:12.032088 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.032075    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd\": container with ID starting with 77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd not found: ID does not exist" containerID="77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"
Jul 28 16:01:12.032147 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.032093    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd"} err="failed to get container status \"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd\": rpc error: code = NotFound desc = could not find container \"77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd\": container with ID starting with 77f7f70dd14c2a661292234911e380bc93bfd62060c0e57f8982717dc9be08bd not found: ID does not exist"
Jul 28 16:01:12.032147 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.032107    2580 scope.go:117] "RemoveContainer" containerID="b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"
Jul 28 16:01:12.032411 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.032396    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb\": container with ID starting with b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb not found: ID does not exist" containerID="b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"
Jul 28 16:01:12.032454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.032414    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb"} err="failed to get container status \"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb\": rpc error: code = NotFound desc = could not find container \"b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb\": container with ID starting with b956aeb8369772772c13ed954b35eb767a7bc42701dd19b16c2c12cd9b8a5ceb not found: ID does not exist"
Jul 28 16:01:12.032454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.032428    2580 scope.go:117] "RemoveContainer" containerID="e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"
Jul 28 16:01:12.037746 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.037724    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kbtvt"]
Jul 28 16:01:12.040615 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.040600    2580 scope.go:117] "RemoveContainer" containerID="9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"
Jul 28 16:01:12.049137 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.049107    2580 scope.go:117] "RemoveContainer" containerID="f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"
Jul 28 16:01:12.053743 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.053717    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 16:01:12.057369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.057345    2580 scope.go:117] "RemoveContainer" containerID="e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"
Jul 28 16:01:12.057650 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.057631    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456\": container with ID starting with e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456 not found: ID does not exist" containerID="e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"
Jul 28 16:01:12.057711 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.057656    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456"} err="failed to get container status \"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456\": rpc error: code = NotFound desc = could not find container \"e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456\": container with ID starting with e8a2c7bd655acb4e5cddd8d27d63e2ab71344aa0447e197092b424a92007d456 not found: ID does not exist"
Jul 28 16:01:12.057711 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.057675    2580 scope.go:117] "RemoveContainer" containerID="9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"
Jul 28 16:01:12.057905 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.057892    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b\": container with ID starting with 9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b not found: ID does not exist" containerID="9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"
Jul 28 16:01:12.057951 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.057908    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b"} err="failed to get container status \"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b\": rpc error: code = NotFound desc = could not find container \"9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b\": container with ID starting with 9140271bc4ae2522bbde9fbf484fc627b5b501257826d3320a68f39dfe2ddb9b not found: ID does not exist"
Jul 28 16:01:12.057951 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.057926    2580 scope.go:117] "RemoveContainer" containerID="f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"
Jul 28 16:01:12.058161 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:01:12.058139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d\": container with ID starting with f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d not found: ID does not exist" containerID="f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"
Jul 28 16:01:12.058209 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058172    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d"} err="failed to get container status \"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d\": rpc error: code = NotFound desc = could not find container \"f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d\": container with ID starting with f3392237d71f18ae4f64607840373e290aa77e1f42db8aa1cea7f27f0329b85d not found: ID does not exist"
Jul 28 16:01:12.058358 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058342    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-btngv"]
Jul 28 16:01:12.058397 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058485 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.058535 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.058535 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058639 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbrr\" (UniqueName: \"kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.058639 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcsg\" (UniqueName: \"kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058639 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.058774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.058774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.058925 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27gw\" (UniqueName: \"kubernetes.io/projected/27c6860b-d806-4d94-9c84-39777f2f2638-kube-api-access-q27gw\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:12.058925 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.058848    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27c6860b-d806-4d94-9c84-39777f2f2638-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:01:12.059178 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.059076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.059178 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.059150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.059400 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.059287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.059564 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.059545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.059624 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.059602    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.061047 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.061032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.061173 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.061158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.067904 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.067877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcsg\" (UniqueName: \"kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg\") pod \"isvc-canary-promote-v2-predictor-86766c4958-pszr7\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.068496 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.068480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbrr\" (UniqueName: \"kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr\") pod \"isvc-canary-promote-predictor-d9759ffd5-2q9cs\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.112650 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.112621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:01:12.130541 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.130514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:12.141175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.141152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:12.190320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.190260    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" path="/var/lib/kubelet/pods/27c6860b-d806-4d94-9c84-39777f2f2638/volumes"
Jul 28 16:01:12.191151 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.190998    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617cba57-d003-44d8-8c64-ba0e7db44155" path="/var/lib/kubelet/pods/617cba57-d003-44d8-8c64-ba0e7db44155/volumes"
Jul 28 16:01:12.280462 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.280440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:01:12.282586 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:01:12.282551    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dc6858_99f6_4e1f_8048_e0d39edfb091.slice/crio-6df36dd3f825c0a304461ed9b7f93a6eae93aedefb0ee8c5d2ab9e5b6f3eb8f4 WatchSource:0}: Error finding container 6df36dd3f825c0a304461ed9b7f93a6eae93aedefb0ee8c5d2ab9e5b6f3eb8f4: Status 404 returned error can't find the container with id 6df36dd3f825c0a304461ed9b7f93a6eae93aedefb0ee8c5d2ab9e5b6f3eb8f4
Jul 28 16:01:12.298154 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:12.298112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:01:12.300487 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:01:12.300457    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5754f5a4_d588_438b_b6a8_e58b4ea75778.slice/crio-926dfd653a400db9a5e2781b8348052926e3608426b2edcd64c17d73aa27ee45 WatchSource:0}: Error finding container 926dfd653a400db9a5e2781b8348052926e3608426b2edcd64c17d73aa27ee45: Status 404 returned error can't find the container with id 926dfd653a400db9a5e2781b8348052926e3608426b2edcd64c17d73aa27ee45
Jul 28 16:01:13.014914 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:13.014862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerStarted","Data":"bbd031fc2d10f0a2725c35f8c000153e78b7268aa2a72919303a3a1e59883521"}
Jul 28 16:01:13.015394 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:13.014979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerStarted","Data":"926dfd653a400db9a5e2781b8348052926e3608426b2edcd64c17d73aa27ee45"}
Jul 28 16:01:13.016517 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:13.016487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerStarted","Data":"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"}
Jul 28 16:01:13.016634 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:13.016554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerStarted","Data":"6df36dd3f825c0a304461ed9b7f93a6eae93aedefb0ee8c5d2ab9e5b6f3eb8f4"}
Jul 28 16:01:17.039571 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:17.039531    2580 generic.go:356] "Generic (PLEG): container finished" podID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerID="bbd031fc2d10f0a2725c35f8c000153e78b7268aa2a72919303a3a1e59883521" exitCode=0
Jul 28 16:01:17.040042 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:17.039605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerDied","Data":"bbd031fc2d10f0a2725c35f8c000153e78b7268aa2a72919303a3a1e59883521"}
Jul 28 16:01:17.041141 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:17.041105    2580 generic.go:356] "Generic (PLEG): container finished" podID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerID="7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b" exitCode=0
Jul 28 16:01:17.041247 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:17.041181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerDied","Data":"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"}
Jul 28 16:01:18.047665 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.047628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerStarted","Data":"59152ef362cb54f7939edbd80451d1ed3050768344bc69e82a42a60552f694f8"}
Jul 28 16:01:18.048112 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.047706    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerStarted","Data":"e5dd249d9249397c82af34e3b67a6a86151da7400a218c7efb4af06a6bc48263"}
Jul 28 16:01:18.048112 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.048020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:18.048266 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.048162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:18.049479 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.049448    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:18.049781 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.049762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerStarted","Data":"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"}
Jul 28 16:01:18.049837 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.049812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerStarted","Data":"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"}
Jul 28 16:01:18.050006 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.049990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:18.119489 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.119433    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podStartSLOduration=3.240811531 podStartE2EDuration="7.119417142s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878605611s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:18.115328918 +0000 UTC m=+1692.483981836" watchObservedRunningTime="2026-07-28 16:01:18.119417142 +0000 UTC m=+1692.488070055"
Jul 28 16:01:18.119645 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:18.119542    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podStartSLOduration=3.069226299 podStartE2EDuration="7.11953529s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.050308991s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:18.082753197 +0000 UTC m=+1692.451406113" watchObservedRunningTime="2026-07-28 16:01:18.11953529 +0000 UTC m=+1692.488188206"
Jul 28 16:01:19.055007 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:19.054965    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:19.055432 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:19.055078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:19.056333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:19.056309    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:01:20.059840 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:20.059795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:01:24.059896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:24.059865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:01:24.060378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:24.060319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:25.064401 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:25.064366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:01:25.065053 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:25.065023    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:01:34.061386 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:34.061295    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:35.065698 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:35.065653    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:01:44.060317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:44.060271    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:45.065082 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:45.065038    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:01:54.061011 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:54.060969    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:01:55.065632 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:01:55.065583    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:02:04.060484 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:04.060442    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:02:05.065526 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:05.065485    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:02:14.060719 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:14.060676    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:02:15.065828 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:15.065787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:02:24.060615 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:24.060571    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:02:25.065542 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:25.065492    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:02:34.061281 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:34.061251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:02:35.065301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:35.065271    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:02:38.895438 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:38.895380    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:41.888108 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:41.888071    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:02:41.888512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:41.888297    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" containerID="cri-o://b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03" gracePeriod=30
Jul 28 16:02:41.888512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:41.888356    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kube-rbac-proxy" containerID="cri-o://2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1" gracePeriod=30
Jul 28 16:02:41.911857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:41.911823    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:02:42.467708 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:42.467670    2580 generic.go:356] "Generic (PLEG): container finished" podID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerID="2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1" exitCode=2
Jul 28 16:02:42.467881 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:42.467705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerDied","Data":"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"}
Jul 28 16:02:45.060672 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:45.060617    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 16:02:45.065082 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:45.065048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:02:46.641942 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.641917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:02:46.775320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location\") pod \"51dc6858-99f6-4e1f-8048-e0d39edfb091\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") "
Jul 28 16:02:46.775320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config\") pod \"51dc6858-99f6-4e1f-8048-e0d39edfb091\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") "
Jul 28 16:02:46.775509 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert\") pod \"51dc6858-99f6-4e1f-8048-e0d39edfb091\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") "
Jul 28 16:02:46.775509 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls\") pod \"51dc6858-99f6-4e1f-8048-e0d39edfb091\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") "
Jul 28 16:02:46.775509 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brbrr\" (UniqueName: \"kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr\") pod \"51dc6858-99f6-4e1f-8048-e0d39edfb091\" (UID: \"51dc6858-99f6-4e1f-8048-e0d39edfb091\") "
Jul 28 16:02:46.775668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51dc6858-99f6-4e1f-8048-e0d39edfb091" (UID: "51dc6858-99f6-4e1f-8048-e0d39edfb091"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:02:46.775750 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775720    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51dc6858-99f6-4e1f-8048-e0d39edfb091" (UID: "51dc6858-99f6-4e1f-8048-e0d39edfb091"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:02:46.775815 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.775780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51dc6858-99f6-4e1f-8048-e0d39edfb091" (UID: "51dc6858-99f6-4e1f-8048-e0d39edfb091"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:02:46.777649 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.777621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr" (OuterVolumeSpecName: "kube-api-access-brbrr") pod "51dc6858-99f6-4e1f-8048-e0d39edfb091" (UID: "51dc6858-99f6-4e1f-8048-e0d39edfb091"). InnerVolumeSpecName "kube-api-access-brbrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:02:46.777649 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.777634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51dc6858-99f6-4e1f-8048-e0d39edfb091" (UID: "51dc6858-99f6-4e1f-8048-e0d39edfb091"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:02:46.876422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.876383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:02:46.876422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.876417    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51dc6858-99f6-4e1f-8048-e0d39edfb091-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:02:46.876422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.876428    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51dc6858-99f6-4e1f-8048-e0d39edfb091-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:02:46.876659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.876439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brbrr\" (UniqueName: \"kubernetes.io/projected/51dc6858-99f6-4e1f-8048-e0d39edfb091-kube-api-access-brbrr\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:02:46.876659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:46.876447    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51dc6858-99f6-4e1f-8048-e0d39edfb091-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:02:47.493194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.493154    2580 generic.go:356] "Generic (PLEG): container finished" podID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerID="b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03" exitCode=0
Jul 28 16:02:47.493369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.493200    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"
Jul 28 16:02:47.493369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.493204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerDied","Data":"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"}
Jul 28 16:02:47.493369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.493342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs" event={"ID":"51dc6858-99f6-4e1f-8048-e0d39edfb091","Type":"ContainerDied","Data":"6df36dd3f825c0a304461ed9b7f93a6eae93aedefb0ee8c5d2ab9e5b6f3eb8f4"}
Jul 28 16:02:47.493471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.493385    2580 scope.go:117] "RemoveContainer" containerID="2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"
Jul 28 16:02:47.502538 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.502521    2580 scope.go:117] "RemoveContainer" containerID="b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"
Jul 28 16:02:47.510614 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.510428    2580 scope.go:117] "RemoveContainer" containerID="7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"
Jul 28 16:02:47.517260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.517232    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:02:47.518591 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.518576    2580 scope.go:117] "RemoveContainer" containerID="2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"
Jul 28 16:02:47.518868 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:02:47.518849    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1\": container with ID starting with 2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1 not found: ID does not exist" containerID="2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"
Jul 28 16:02:47.518917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.518881    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1"} err="failed to get container status \"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1\": rpc error: code = NotFound desc = could not find container \"2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1\": container with ID starting with 2e55cc004e9fd3e5323546e5d3a5f44cb394d05e539de0c267517ee3dfe8a6b1 not found: ID does not exist"
Jul 28 16:02:47.518917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.518898    2580 scope.go:117] "RemoveContainer" containerID="b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"
Jul 28 16:02:47.519110 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:02:47.519095    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03\": container with ID starting with b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03 not found: ID does not exist" containerID="b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"
Jul 28 16:02:47.519185 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.519115    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03"} err="failed to get container status \"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03\": rpc error: code = NotFound desc = could not find container \"b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03\": container with ID starting with b96237bd918909a924305ae54c4cd2e78f06ab669538ac9034771991dd530a03 not found: ID does not exist"
Jul 28 16:02:47.519185 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.519145    2580 scope.go:117] "RemoveContainer" containerID="7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"
Jul 28 16:02:47.519387 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:02:47.519368    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b\": container with ID starting with 7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b not found: ID does not exist" containerID="7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"
Jul 28 16:02:47.519432 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.519393    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b"} err="failed to get container status \"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b\": rpc error: code = NotFound desc = could not find container \"7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b\": container with ID starting with 7c63ced2bd8241b2a4e198577762be3c24b6d4883dcb3d6b8420842422d8eb4b not found: ID does not exist"
Jul 28 16:02:47.523540 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:47.523521    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-2q9cs"]
Jul 28 16:02:48.186499 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:02:48.186458    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" path="/var/lib/kubelet/pods/51dc6858-99f6-4e1f-8048-e0d39edfb091/volumes"
Jul 28 16:04:32.199810 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.199730    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:04:32.202400 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.199979    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" containerID="cri-o://e5dd249d9249397c82af34e3b67a6a86151da7400a218c7efb4af06a6bc48263" gracePeriod=30
Jul 28 16:04:32.202400 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.200020    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kube-rbac-proxy" containerID="cri-o://59152ef362cb54f7939edbd80451d1ed3050768344bc69e82a42a60552f694f8" gracePeriod=30
Jul 28 16:04:32.265756 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.265722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:32.964755 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.964664    2580 generic.go:356] "Generic (PLEG): container finished" podID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerID="59152ef362cb54f7939edbd80451d1ed3050768344bc69e82a42a60552f694f8" exitCode=2
Jul 28 16:04:32.964755 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:32.964703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerDied","Data":"59152ef362cb54f7939edbd80451d1ed3050768344bc69e82a42a60552f694f8"}
Jul 28 16:04:34.055840 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:34.055798    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 16:04:34.061232 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:34.061203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:04:36.986903 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:36.986708    2580 generic.go:356] "Generic (PLEG): container finished" podID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerID="e5dd249d9249397c82af34e3b67a6a86151da7400a218c7efb4af06a6bc48263" exitCode=0
Jul 28 16:04:36.986903 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:36.986813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerDied","Data":"e5dd249d9249397c82af34e3b67a6a86151da7400a218c7efb4af06a6bc48263"}
Jul 28 16:04:37.046241 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.046216    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:04:37.067676 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztcsg\" (UniqueName: \"kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg\") pod \"5754f5a4-d588-438b-b6a8-e58b4ea75778\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") "
Jul 28 16:04:37.067851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location\") pod \"5754f5a4-d588-438b-b6a8-e58b4ea75778\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") "
Jul 28 16:04:37.067851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config\") pod \"5754f5a4-d588-438b-b6a8-e58b4ea75778\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") "
Jul 28 16:04:37.067851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls\") pod \"5754f5a4-d588-438b-b6a8-e58b4ea75778\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") "
Jul 28 16:04:37.067851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert\") pod \"5754f5a4-d588-438b-b6a8-e58b4ea75778\" (UID: \"5754f5a4-d588-438b-b6a8-e58b4ea75778\") "
Jul 28 16:04:37.068186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.067993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5754f5a4-d588-438b-b6a8-e58b4ea75778" (UID: "5754f5a4-d588-438b-b6a8-e58b4ea75778"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:04:37.068186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.068121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5754f5a4-d588-438b-b6a8-e58b4ea75778" (UID: "5754f5a4-d588-438b-b6a8-e58b4ea75778"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:37.068345 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.068261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5754f5a4-d588-438b-b6a8-e58b4ea75778" (UID: "5754f5a4-d588-438b-b6a8-e58b4ea75778"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:37.070331 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.070302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg" (OuterVolumeSpecName: "kube-api-access-ztcsg") pod "5754f5a4-d588-438b-b6a8-e58b4ea75778" (UID: "5754f5a4-d588-438b-b6a8-e58b4ea75778"). InnerVolumeSpecName "kube-api-access-ztcsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:37.070601 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.070580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5754f5a4-d588-438b-b6a8-e58b4ea75778" (UID: "5754f5a4-d588-438b-b6a8-e58b4ea75778"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:37.168997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.168902    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5754f5a4-d588-438b-b6a8-e58b4ea75778-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:04:37.168997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.168933    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:04:37.168997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.168942    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5754f5a4-d588-438b-b6a8-e58b4ea75778-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:04:37.168997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.168953    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5754f5a4-d588-438b-b6a8-e58b4ea75778-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:04:37.168997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.168961    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcsg\" (UniqueName: \"kubernetes.io/projected/5754f5a4-d588-438b-b6a8-e58b4ea75778-kube-api-access-ztcsg\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:04:37.255840 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.255803    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:37.307853 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.307818    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:04:37.308179 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308166    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="storage-initializer"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="storage-initializer"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308192    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308197    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308208    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308214    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308224    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308230    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308242    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="storage-initializer"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308247    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="storage-initializer"
Jul 28 16:04:37.308251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="storage-initializer"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="storage-initializer"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308275    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308280    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308292    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308345    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308352    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51dc6858-99f6-4e1f-8048-e0d39edfb091" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kube-rbac-proxy"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308372    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kserve-container"
Jul 28 16:04:37.308642 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.308376    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c6860b-d806-4d94-9c84-39777f2f2638" containerName="kube-rbac-proxy"
Jul 28 16:04:37.311327 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.311312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.314146 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.314102    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 16:04:37.314283 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.314186    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 16:04:37.320943 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.320917    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:04:37.333577 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.333534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:37.369960 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.369925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.370175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.369998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.370175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.370022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.370175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.370041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4sq\" (UniqueName: \"kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.370175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.370061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.389493 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.389463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:37.471435 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.471435 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.471435 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4sq\" (UniqueName: \"kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.471748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.471748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.471748 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:04:37.471552    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 16:04:37.471748 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:04:37.471620    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls podName:8b33c035-9ad6-4da4-ae73-edc1042c6c19 nodeName:}" failed. No retries permitted until 2026-07-28 16:04:37.971604644 +0000 UTC m=+1892.340257538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 16:04:37.471981 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.471837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.472048 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.472018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.472220 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.472199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.482729 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.482708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4sq\" (UniqueName: \"kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.663270 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.663221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:04:37.976664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.976620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.979188 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.979160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:37.992521 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.992495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7" event={"ID":"5754f5a4-d588-438b-b6a8-e58b4ea75778","Type":"ContainerDied","Data":"926dfd653a400db9a5e2781b8348052926e3608426b2edcd64c17d73aa27ee45"}
Jul 28 16:04:37.992521 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.992509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"
Jul 28 16:04:37.992985 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:37.992557    2580 scope.go:117] "RemoveContainer" containerID="59152ef362cb54f7939edbd80451d1ed3050768344bc69e82a42a60552f694f8"
Jul 28 16:04:38.002113 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.002095    2580 scope.go:117] "RemoveContainer" containerID="e5dd249d9249397c82af34e3b67a6a86151da7400a218c7efb4af06a6bc48263"
Jul 28 16:04:38.009925 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.009906    2580 scope.go:117] "RemoveContainer" containerID="bbd031fc2d10f0a2725c35f8c000153e78b7268aa2a72919303a3a1e59883521"
Jul 28 16:04:38.016057 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.016025    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:04:38.023196 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.023167    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-pszr7"]
Jul 28 16:04:38.191259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.191222    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5754f5a4-d588-438b-b6a8-e58b4ea75778" path="/var/lib/kubelet/pods/5754f5a4-d588-438b-b6a8-e58b4ea75778/volumes"
Jul 28 16:04:38.222446 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.222411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:38.358521 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.358445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:04:38.361006 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:04:38.360975    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33c035_9ad6_4da4_ae73_edc1042c6c19.slice/crio-1b494eb6d810d965e5d5d4361c8ba5bfa08c1f699dc7cc7d9befcaa08f72635a WatchSource:0}: Error finding container 1b494eb6d810d965e5d5d4361c8ba5bfa08c1f699dc7cc7d9befcaa08f72635a: Status 404 returned error can't find the container with id 1b494eb6d810d965e5d5d4361c8ba5bfa08c1f699dc7cc7d9befcaa08f72635a
Jul 28 16:04:38.999334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.999298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerStarted","Data":"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"}
Jul 28 16:04:38.999803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:38.999366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerStarted","Data":"1b494eb6d810d965e5d5d4361c8ba5bfa08c1f699dc7cc7d9befcaa08f72635a"}
Jul 28 16:04:43.020829 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:43.020796    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerID="d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90" exitCode=0
Jul 28 16:04:43.021249 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:43.020869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerDied","Data":"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"}
Jul 28 16:04:44.027896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:44.027863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerStarted","Data":"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"}
Jul 28 16:04:44.028340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:44.027920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerStarted","Data":"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"}
Jul 28 16:04:44.028692 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:44.028670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:44.058409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:44.058358    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podStartSLOduration=3.347546007 podStartE2EDuration="7.058341149s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.710795142s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:44.057737075 +0000 UTC m=+1898.426389992" watchObservedRunningTime="2026-07-28 16:04:44.058341149 +0000 UTC m=+1898.426994065"
Jul 28 16:04:45.032997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:45.032962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:45.034158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:45.034112    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:04:46.037030 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:46.036994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:04:51.042535 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:51.042510    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:04:51.042991 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:04:51.042965    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:01.043263 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:01.043218    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:11.043753 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:11.043711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:21.043763 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:21.043725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:31.043015 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:31.042973    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:41.043888 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:41.043850    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:05:51.042983 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:05:51.042934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:06:01.044419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:06:01.044325    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:06:01.601434 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:06:01.601395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:06:07.360301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:06:07.360268    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:37.562267 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.562191    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:07:37.562760 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.562459    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" containerID="cri-o://d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a" gracePeriod=30
Jul 28 16:07:37.562760 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.562517    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kube-rbac-proxy" containerID="cri-o://ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679" gracePeriod=30
Jul 28 16:07:37.618744 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.618710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:37.780137 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.780087    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerID="ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679" exitCode=2
Jul 28 16:07:37.780301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:37.780162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerDied","Data":"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"}
Jul 28 16:07:41.037787 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:41.037751    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 16:07:41.043095 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:41.043070    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:07:42.418096 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.418071    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:07:42.510422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert\") pod \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") "
Jul 28 16:07:42.510422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510377    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4sq\" (UniqueName: \"kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq\") pod \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") "
Jul 28 16:07:42.510422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location\") pod \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") "
Jul 28 16:07:42.510628 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") pod \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") "
Jul 28 16:07:42.510747 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b33c035-9ad6-4da4-ae73-edc1042c6c19" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:07:42.510792 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b33c035-9ad6-4da4-ae73-edc1042c6c19" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:07:42.510835 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510821    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:07:42.510869 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.510840    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:07:42.512600 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.512575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq" (OuterVolumeSpecName: "kube-api-access-wf4sq") pod "8b33c035-9ad6-4da4-ae73-edc1042c6c19" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19"). InnerVolumeSpecName "kube-api-access-wf4sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:07:42.512695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.512672    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b33c035-9ad6-4da4-ae73-edc1042c6c19" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:07:42.611433 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.611398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config\") pod \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\" (UID: \"8b33c035-9ad6-4da4-ae73-edc1042c6c19\") "
Jul 28 16:07:42.611584 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.611556    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4sq\" (UniqueName: \"kubernetes.io/projected/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-api-access-wf4sq\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:07:42.611584 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.611567    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b33c035-9ad6-4da4-ae73-edc1042c6c19-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:07:42.611774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.611749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b33c035-9ad6-4da4-ae73-edc1042c6c19" (UID: "8b33c035-9ad6-4da4-ae73-edc1042c6c19"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:07:42.712837 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.712804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b33c035-9ad6-4da4-ae73-edc1042c6c19-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:07:42.714091 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.714045    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:42.793587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.793481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:42.797950 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.797916    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:07:42.798276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798264    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kube-rbac-proxy"
Jul 28 16:07:42.798276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798278    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kube-rbac-proxy"
Jul 28 16:07:42.798357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798293    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="storage-initializer"
Jul 28 16:07:42.798357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798298    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="storage-initializer"
Jul 28 16:07:42.798357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798307    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container"
Jul 28 16:07:42.798357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798312    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container"
Jul 28 16:07:42.798477 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798371    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kube-rbac-proxy"
Jul 28 16:07:42.798477 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.798380    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerName="kserve-container"
Jul 28 16:07:42.801719 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.801702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.804356 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.804327    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 16:07:42.804500 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.804404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 16:07:42.807194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.807169    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" containerID="d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a" exitCode=0
Jul 28 16:07:42.807317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.807228    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"
Jul 28 16:07:42.807317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.807245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerDied","Data":"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"}
Jul 28 16:07:42.807317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.807300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8" event={"ID":"8b33c035-9ad6-4da4-ae73-edc1042c6c19","Type":"ContainerDied","Data":"1b494eb6d810d965e5d5d4361c8ba5bfa08c1f699dc7cc7d9befcaa08f72635a"}
Jul 28 16:07:42.807487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.807325    2580 scope.go:117] "RemoveContainer" containerID="ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"
Jul 28 16:07:42.814194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.814167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.814316 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.814207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.814316 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.814274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.814316 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.814303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.814477 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.814387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmts\" (UniqueName: \"kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.815884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.815862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:07:42.818766 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.818749    2580 scope.go:117] "RemoveContainer" containerID="d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"
Jul 28 16:07:42.828011 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.827532    2580 scope.go:117] "RemoveContainer" containerID="d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"
Jul 28 16:07:42.835990 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.835971    2580 scope.go:117] "RemoveContainer" containerID="ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"
Jul 28 16:07:42.836314 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:07:42.836293    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679\": container with ID starting with ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679 not found: ID does not exist" containerID="ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"
Jul 28 16:07:42.836380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.836325    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679"} err="failed to get container status \"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679\": rpc error: code = NotFound desc = could not find container \"ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679\": container with ID starting with ca3ed1ec6fe9505ceeb2a2fd4b49351c338bbf68e81c8923bf8a4e77aa62b679 not found: ID does not exist"
Jul 28 16:07:42.836380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.836343    2580 scope.go:117] "RemoveContainer" containerID="d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"
Jul 28 16:07:42.836615 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:07:42.836596    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a\": container with ID starting with d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a not found: ID does not exist" containerID="d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"
Jul 28 16:07:42.836668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.836625    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a"} err="failed to get container status \"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a\": rpc error: code = NotFound desc = could not find container \"d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a\": container with ID starting with d8d8013de5c3c1579c354aff92025b354fc92f63754f81e1a78214033a17bf5a not found: ID does not exist"
Jul 28 16:07:42.836668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.836645    2580 scope.go:117] "RemoveContainer" containerID="d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"
Jul 28 16:07:42.836863 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:07:42.836834    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90\": container with ID starting with d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90 not found: ID does not exist" containerID="d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"
Jul 28 16:07:42.836903 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.836871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90"} err="failed to get container status \"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90\": rpc error: code = NotFound desc = could not find container \"d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90\": container with ID starting with d140b694256275536977e34ed1191ef3529e76a0a6228f79a6000594a352cc90 not found: ID does not exist"
Jul 28 16:07:42.845579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.845559    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:07:42.848400 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.848381    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-rcwz8"]
Jul 28 16:07:42.914919 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.914887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915037 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.914928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915037 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.914989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmts\" (UniqueName: \"kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915037 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.915021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915319 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.915047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915319 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:07:42.915112    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 16:07:42.915319 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:07:42.915207    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls podName:d9a86f43-519c-45e5-9b9c-c62884df0c27 nodeName:}" failed. No retries permitted until 2026-07-28 16:07:43.415185161 +0000 UTC m=+2077.783838071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls") pod "isvc-canary-stop-v2-predictor-77f778d864-2zcgs" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 28 16:07:42.915471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.915452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.915581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.915664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.915647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:42.924551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:42.924525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmts\" (UniqueName: \"kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:43.168537 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.168499    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:43.392916 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.392879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:07:43.419309 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.419239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:43.421829 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.421808    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-2zcgs\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:43.714632 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.714541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:43.851206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:43.851180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:07:43.853374 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:07:43.853346    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a86f43_519c_45e5_9b9c_c62884df0c27.slice/crio-306fec10569e8020e37d5eaec432321dde1678ab86c70f62b2ca1622e96912f1 WatchSource:0}: Error finding container 306fec10569e8020e37d5eaec432321dde1678ab86c70f62b2ca1622e96912f1: Status 404 returned error can't find the container with id 306fec10569e8020e37d5eaec432321dde1678ab86c70f62b2ca1622e96912f1
Jul 28 16:07:44.187048 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:44.187002    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b33c035-9ad6-4da4-ae73-edc1042c6c19" path="/var/lib/kubelet/pods/8b33c035-9ad6-4da4-ae73-edc1042c6c19/volumes"
Jul 28 16:07:44.818957 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:44.818917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerStarted","Data":"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"}
Jul 28 16:07:44.819357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:44.818986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerStarted","Data":"306fec10569e8020e37d5eaec432321dde1678ab86c70f62b2ca1622e96912f1"}
Jul 28 16:07:47.834442 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:47.834403    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerID="cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4" exitCode=0
Jul 28 16:07:47.834819 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:47.834478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerDied","Data":"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"}
Jul 28 16:07:48.842888 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:48.842847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerStarted","Data":"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"}
Jul 28 16:07:48.843318 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:48.842913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerStarted","Data":"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"}
Jul 28 16:07:48.843318 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:48.843111    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:48.865572 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:48.865517    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podStartSLOduration=3.030633416 podStartE2EDuration="6.865499353s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.834865937s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:48.863026807 +0000 UTC m=+2083.231679750" watchObservedRunningTime="2026-07-28 16:07:48.865499353 +0000 UTC m=+2083.234152268"
Jul 28 16:07:49.848120 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:49.848080    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:49.849170 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:49.849118    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:07:50.852398 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:50.852360    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:07:55.856626 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:55.856598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:07:55.857110 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:07:55.857082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:05.857354 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:05.857313    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:15.857360 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:15.857317    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:25.857185 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:25.857140    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:35.857565 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:35.857517    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:45.857524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:45.857485    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:08:55.857202 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:08:55.857161    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:09:05.182282 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:05.182252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:09:12.792391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:12.792346    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:12.815831 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:12.815794    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:09:12.816095 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:12.816067    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" containerID="cri-o://268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f" gracePeriod=30
Jul 28 16:09:12.816199 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:12.816147    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kube-rbac-proxy" containerID="cri-o://e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef" gracePeriod=30
Jul 28 16:09:12.894478 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:12.894440    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:13.203582 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:13.203544    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerID="e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef" exitCode=2
Jul 28 16:09:13.203778 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:13.203619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerDied","Data":"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"}
Jul 28 16:09:15.182375 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:15.182332    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:09:15.852582 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:15.852543    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 16:09:17.762425 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.762400    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:09:17.868993 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.868958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") pod \"d9a86f43-519c-45e5-9b9c-c62884df0c27\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") "
Jul 28 16:09:17.869218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bmts\" (UniqueName: \"kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts\") pod \"d9a86f43-519c-45e5-9b9c-c62884df0c27\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") "
Jul 28 16:09:17.869218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869051    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location\") pod \"d9a86f43-519c-45e5-9b9c-c62884df0c27\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") "
Jul 28 16:09:17.869218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert\") pod \"d9a86f43-519c-45e5-9b9c-c62884df0c27\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") "
Jul 28 16:09:17.869391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config\") pod \"d9a86f43-519c-45e5-9b9c-c62884df0c27\" (UID: \"d9a86f43-519c-45e5-9b9c-c62884df0c27\") "
Jul 28 16:09:17.869456 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9a86f43-519c-45e5-9b9c-c62884df0c27" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:09:17.869517 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869487    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9a86f43-519c-45e5-9b9c-c62884df0c27" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:09:17.869618 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9a86f43-519c-45e5-9b9c-c62884df0c27" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:09:17.869715 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869701    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9a86f43-519c-45e5-9b9c-c62884df0c27-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:09:17.869770 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869723    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:09:17.869770 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.869741    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:09:17.871362 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.871339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts" (OuterVolumeSpecName: "kube-api-access-7bmts") pod "d9a86f43-519c-45e5-9b9c-c62884df0c27" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27"). InnerVolumeSpecName "kube-api-access-7bmts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:09:17.871362 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.871355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9a86f43-519c-45e5-9b9c-c62884df0c27" (UID: "d9a86f43-519c-45e5-9b9c-c62884df0c27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:09:17.971205 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.971161    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9a86f43-519c-45e5-9b9c-c62884df0c27-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:09:17.971205 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:17.971204    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bmts\" (UniqueName: \"kubernetes.io/projected/d9a86f43-519c-45e5-9b9c-c62884df0c27-kube-api-access-7bmts\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:09:18.227235 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.227198    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerID="268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f" exitCode=0
Jul 28 16:09:18.227391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.227239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerDied","Data":"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"}
Jul 28 16:09:18.227391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.227276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"
Jul 28 16:09:18.227391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.227292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs" event={"ID":"d9a86f43-519c-45e5-9b9c-c62884df0c27","Type":"ContainerDied","Data":"306fec10569e8020e37d5eaec432321dde1678ab86c70f62b2ca1622e96912f1"}
Jul 28 16:09:18.227391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.227319    2580 scope.go:117] "RemoveContainer" containerID="e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"
Jul 28 16:09:18.238579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.238503    2580 scope.go:117] "RemoveContainer" containerID="268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"
Jul 28 16:09:18.246600 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.246504    2580 scope.go:117] "RemoveContainer" containerID="cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"
Jul 28 16:09:18.249302 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.249281    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:09:18.254722 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.254692    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-2zcgs"]
Jul 28 16:09:18.256106 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256085    2580 scope.go:117] "RemoveContainer" containerID="e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"
Jul 28 16:09:18.256423 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:09:18.256394    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef\": container with ID starting with e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef not found: ID does not exist" containerID="e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"
Jul 28 16:09:18.256528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256432    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef"} err="failed to get container status \"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef\": rpc error: code = NotFound desc = could not find container \"e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef\": container with ID starting with e95394c221ec3e87bd9d1359e8d89c8110fe12430ccab0f30ba80791d1090fef not found: ID does not exist"
Jul 28 16:09:18.256528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256452    2580 scope.go:117] "RemoveContainer" containerID="268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"
Jul 28 16:09:18.256710 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:09:18.256689    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f\": container with ID starting with 268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f not found: ID does not exist" containerID="268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"
Jul 28 16:09:18.256809 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256718    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f"} err="failed to get container status \"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f\": rpc error: code = NotFound desc = could not find container \"268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f\": container with ID starting with 268b0b9210f43b777ff4ecfbb598df688557f9c62a3aef4c13a1dff0f42fe81f not found: ID does not exist"
Jul 28 16:09:18.256809 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256739    2580 scope.go:117] "RemoveContainer" containerID="cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"
Jul 28 16:09:18.256988 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:09:18.256969    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4\": container with ID starting with cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4 not found: ID does not exist" containerID="cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"
Jul 28 16:09:18.257029 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:18.256995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4"} err="failed to get container status \"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4\": rpc error: code = NotFound desc = could not find container \"cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4\": container with ID starting with cd881f9d8aa7a5211574d027f8baa6020b70c12717730aea1539f9d48f7eaea4 not found: ID does not exist"
Jul 28 16:09:20.186102 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:20.186070    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" path="/var/lib/kubelet/pods/d9a86f43-519c-45e5-9b9c-c62884df0c27/volumes"
Jul 28 16:09:23.028228 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:23.028186    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:09:23.124429 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:09:23.124396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:10:34.648493 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:10:34.648416    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:10:36.274771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:10:36.274714    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:10:53.308305 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:10:53.308267    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:10:58.336097 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:10:58.336057    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:10:58.678248 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:10:58.678118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:28.749546 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:28.749511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:28.811889 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:28.811854    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:33.800233 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.800188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:33.860800 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.860763    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:12:33.861163 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kube-rbac-proxy"
Jul 28 16:12:33.861217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861167    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kube-rbac-proxy"
Jul 28 16:12:33.861217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861188    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="storage-initializer"
Jul 28 16:12:33.861217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="storage-initializer"
Jul 28 16:12:33.861217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861201    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container"
Jul 28 16:12:33.861217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861206    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container"
Jul 28 16:12:33.861380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861266    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kserve-container"
Jul 28 16:12:33.861380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.861276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a86f43-519c-45e5-9b9c-c62884df0c27" containerName="kube-rbac-proxy"
Jul 28 16:12:33.864242 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.864226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.867913 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.867885    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:12:33.868068 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.867887    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:12:33.868068 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.867927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:12:33.868068 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.867935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 16:12:33.868245 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.868230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:12:33.869042 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.869025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:12:33.869172 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.869082    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:12:33.879117 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.879091    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:12:33.883336 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.883303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.883468 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.883342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.883468 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.883373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.883468 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.883444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgpr\" (UniqueName: \"kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.883578 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.883512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.984900 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.984850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.984900 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.984908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985211 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.984930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985211 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.984961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985211 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.984992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgpr\" (UniqueName: \"kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985211 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:12:33.985116    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 28 16:12:33.985211 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:12:33.985212    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls podName:e208551f-a10a-4440-ba73-609783ccd5f3 nodeName:}" failed. No retries permitted until 2026-07-28 16:12:34.485192051 +0000 UTC m=+2368.853844964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 28 16:12:33.985431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.985396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985641 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.985613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.985685 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.985647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:33.997878 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:33.997848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgpr\" (UniqueName: \"kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:34.043681 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:34.043651    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:12:34.488942 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:34.488902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:34.491437 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:34.491405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:34.775010 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:34.774916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:12:34.915915 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:34.915889    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:12:34.918377 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:12:34.918339    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode208551f_a10a_4440_ba73_609783ccd5f3.slice/crio-3bed4115d8dd5c69e3b25c6723973a11a117529d5ecdc623ee57a5e8197b5edd WatchSource:0}: Error finding container 3bed4115d8dd5c69e3b25c6723973a11a117529d5ecdc623ee57a5e8197b5edd: Status 404 returned error can't find the container with id 3bed4115d8dd5c69e3b25c6723973a11a117529d5ecdc623ee57a5e8197b5edd
Jul 28 16:12:35.066844 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:35.066742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerStarted","Data":"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"}
Jul 28 16:12:35.066844 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:35.066815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerStarted","Data":"3bed4115d8dd5c69e3b25c6723973a11a117529d5ecdc623ee57a5e8197b5edd"}
Jul 28 16:12:39.084349 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:39.084317    2580 generic.go:356] "Generic (PLEG): container finished" podID="e208551f-a10a-4440-ba73-609783ccd5f3" containerID="783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d" exitCode=0
Jul 28 16:12:39.084812 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:39.084401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerDied","Data":"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"}
Jul 28 16:12:39.085594 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:12:39.085584    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:14:54.783322 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:14:54.783279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerStarted","Data":"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"}
Jul 28 16:14:54.783768 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:14:54.783362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerStarted","Data":"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"}
Jul 28 16:14:54.783768 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:14:54.783404    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:14:54.829016 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:14:54.828929    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" podStartSLOduration=2.8472754670000002 podStartE2EDuration="2m21.828905419s" totalImagesPullingTime="2m15.082567269s" totalInitContainerRuntime="3.899062683s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:14:54.824862454 +0000 UTC m=+2509.193515371" watchObservedRunningTime="2026-07-28 16:14:54.828905419 +0000 UTC m=+2509.197558336"
Jul 28 16:14:55.788231 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:14:55.788198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:15:01.799551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:01.799519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:15:08.095153 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:08.095100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:31.803308 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:31.803276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:15:34.061039 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.061002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:15:34.061525 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.061316    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kserve-container" containerID="cri-o://c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a" gracePeriod=30
Jul 28 16:15:34.061525 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.061350    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kube-rbac-proxy" containerID="cri-o://acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e" gracePeriod=30
Jul 28 16:15:34.115435 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.115400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:34.957895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.957863    2580 generic.go:356] "Generic (PLEG): container finished" podID="e208551f-a10a-4440-ba73-609783ccd5f3" containerID="acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e" exitCode=2
Jul 28 16:15:34.958049 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:34.957929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerDied","Data":"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"}
Jul 28 16:15:35.143417 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.143391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:15:35.325897 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.325810    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config\") pod \"e208551f-a10a-4440-ba73-609783ccd5f3\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") "
Jul 28 16:15:35.325897 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.325869    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhgpr\" (UniqueName: \"kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr\") pod \"e208551f-a10a-4440-ba73-609783ccd5f3\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") "
Jul 28 16:15:35.326195 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.325900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") pod \"e208551f-a10a-4440-ba73-609783ccd5f3\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") "
Jul 28 16:15:35.326195 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location\") pod \"e208551f-a10a-4440-ba73-609783ccd5f3\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") "
Jul 28 16:15:35.326195 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert\") pod \"e208551f-a10a-4440-ba73-609783ccd5f3\" (UID: \"e208551f-a10a-4440-ba73-609783ccd5f3\") "
Jul 28 16:15:35.326377 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e208551f-a10a-4440-ba73-609783ccd5f3" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:35.326791 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e208551f-a10a-4440-ba73-609783ccd5f3" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:15:35.326908 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326818    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.326908 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.326826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e208551f-a10a-4440-ba73-609783ccd5f3" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:15:35.328281 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.328247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e208551f-a10a-4440-ba73-609783ccd5f3" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:15:35.328397 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.328332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr" (OuterVolumeSpecName: "kube-api-access-qhgpr") pod "e208551f-a10a-4440-ba73-609783ccd5f3" (UID: "e208551f-a10a-4440-ba73-609783ccd5f3"). InnerVolumeSpecName "kube-api-access-qhgpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:15:35.427704 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.427667    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgpr\" (UniqueName: \"kubernetes.io/projected/e208551f-a10a-4440-ba73-609783ccd5f3-kube-api-access-qhgpr\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.427704 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.427698    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e208551f-a10a-4440-ba73-609783ccd5f3-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.427704 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.427712    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e208551f-a10a-4440-ba73-609783ccd5f3-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.427952 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.427721    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e208551f-a10a-4440-ba73-609783ccd5f3-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:15:35.963725 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.963690    2580 generic.go:356] "Generic (PLEG): container finished" podID="e208551f-a10a-4440-ba73-609783ccd5f3" containerID="c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a" exitCode=0
Jul 28 16:15:35.963912 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.963741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerDied","Data":"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"}
Jul 28 16:15:35.963912 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.963765    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"
Jul 28 16:15:35.963912 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.963817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg" event={"ID":"e208551f-a10a-4440-ba73-609783ccd5f3","Type":"ContainerDied","Data":"3bed4115d8dd5c69e3b25c6723973a11a117529d5ecdc623ee57a5e8197b5edd"}
Jul 28 16:15:35.963912 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.963845    2580 scope.go:117] "RemoveContainer" containerID="acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"
Jul 28 16:15:35.972518 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.972503    2580 scope.go:117] "RemoveContainer" containerID="c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"
Jul 28 16:15:35.980120 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.980101    2580 scope.go:117] "RemoveContainer" containerID="783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"
Jul 28 16:15:35.987564 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.987497    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:15:35.988677 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.988659    2580 scope.go:117] "RemoveContainer" containerID="acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"
Jul 28 16:15:35.988967 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:15:35.988944    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e\": container with ID starting with acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e not found: ID does not exist" containerID="acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"
Jul 28 16:15:35.989035 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.988979    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e"} err="failed to get container status \"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e\": rpc error: code = NotFound desc = could not find container \"acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e\": container with ID starting with acc1a8c8af3894056ed289e753caa452fa655b047d020f8bdbfc11b5cd517f5e not found: ID does not exist"
Jul 28 16:15:35.989035 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.988997    2580 scope.go:117] "RemoveContainer" containerID="c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"
Jul 28 16:15:35.989257 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:15:35.989240    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a\": container with ID starting with c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a not found: ID does not exist" containerID="c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"
Jul 28 16:15:35.989328 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.989263    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a"} err="failed to get container status \"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a\": rpc error: code = NotFound desc = could not find container \"c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a\": container with ID starting with c30e94afcf6279b73c72bf222bf752d46069f3adb5a6acf34e7c441d9b05491a not found: ID does not exist"
Jul 28 16:15:35.989328 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.989277    2580 scope.go:117] "RemoveContainer" containerID="783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"
Jul 28 16:15:35.989513 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:15:35.989497    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d\": container with ID starting with 783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d not found: ID does not exist" containerID="783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"
Jul 28 16:15:35.989571 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.989516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d"} err="failed to get container status \"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d\": rpc error: code = NotFound desc = could not find container \"783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d\": container with ID starting with 783bfdbe2d62b0a3aba651a4617b813962e0682e8881ac3bcc9e29909092ef9d not found: ID does not exist"
Jul 28 16:15:35.989771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:35.989750    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-h6dsg"]
Jul 28 16:15:36.186670 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:36.186629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" path="/var/lib/kubelet/pods/e208551f-a10a-4440-ba73-609783ccd5f3/volumes"
Jul 28 16:15:39.174417 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:39.174383    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:41.538682 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:41.538643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:15:59.303609 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:15:59.303556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:16:04.436507 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.436471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:16:04.474905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.474866    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:04.475439 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475421    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="storage-initializer"
Jul 28 16:16:04.475491 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="storage-initializer"
Jul 28 16:16:04.475491 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475476    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kserve-container"
Jul 28 16:16:04.475491 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475485    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kserve-container"
Jul 28 16:16:04.475584 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475498    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kube-rbac-proxy"
Jul 28 16:16:04.475584 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475509    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kube-rbac-proxy"
Jul 28 16:16:04.475650 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475582    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kserve-container"
Jul 28 16:16:04.475650 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.475598    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e208551f-a10a-4440-ba73-609783ccd5f3" containerName="kube-rbac-proxy"
Jul 28 16:16:04.478223 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.478201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.481405 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.481384    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:16:04.481770 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.481755    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:16:04.481770 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.481766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:16:04.482495 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.482478    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:16:04.482595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.482499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:16:04.482595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.482514    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:16:04.482595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.482517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 16:16:04.491289 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.491262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:04.570519 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.570479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.570519 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.570519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.570725 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.570554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.570725 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.570636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.570725 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.570714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4ql\" (UniqueName: \"kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.671923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.671888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.671923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.671926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.671944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.671976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.672024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4ql\" (UniqueName: \"kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672093 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:16:04.672069    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 16:16:04.672296 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:16:04.672173    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls podName:09d238cf-2ef9-4546-a3f7-537c27fea6f4 nodeName:}" failed. No retries permitted until 2026-07-28 16:16:05.172120682 +0000 UTC m=+2579.540773582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 16:16:04.672484 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.672454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672674 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.672651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.672833 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.672811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.682907 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.682875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4ql\" (UniqueName: \"kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:04.798461 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:04.798382    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:16:05.175612 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:05.175571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:05.178119 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:05.178088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:05.389000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:05.388942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:05.527588 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:05.527563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:05.529931 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:16:05.529907    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d238cf_2ef9_4546_a3f7_537c27fea6f4.slice/crio-41dea8f6ec77970222973b7544019e0810ccac3b0583f1d5ce94c6e2db899368 WatchSource:0}: Error finding container 41dea8f6ec77970222973b7544019e0810ccac3b0583f1d5ce94c6e2db899368: Status 404 returned error can't find the container with id 41dea8f6ec77970222973b7544019e0810ccac3b0583f1d5ce94c6e2db899368
Jul 28 16:16:06.093610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:06.093560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerStarted","Data":"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"}
Jul 28 16:16:06.093783 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:06.093680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerStarted","Data":"41dea8f6ec77970222973b7544019e0810ccac3b0583f1d5ce94c6e2db899368"}
Jul 28 16:16:10.112342 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:10.112308    2580 generic.go:356] "Generic (PLEG): container finished" podID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerID="57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b" exitCode=0
Jul 28 16:16:10.112727 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:10.112381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerDied","Data":"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"}
Jul 28 16:16:11.119578 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:11.119539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerStarted","Data":"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"}
Jul 28 16:16:11.120087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:11.119604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerStarted","Data":"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"}
Jul 28 16:16:11.120087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:11.119819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:11.120087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:11.119865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:11.144660 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:11.144601    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" podStartSLOduration=2.821429616 podStartE2EDuration="7.144585749s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.323156133s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:16:11.142696933 +0000 UTC m=+2585.511349851" watchObservedRunningTime="2026-07-28 16:16:11.144585749 +0000 UTC m=+2585.513238665"
Jul 28 16:16:17.129332 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:17.129293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:47.133568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:47.133533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:54.573206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:54.573158    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:54.573682 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:54.573464    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kserve-container" containerID="cri-o://a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4" gracePeriod=30
Jul 28 16:16:54.573682 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:54.573525    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kube-rbac-proxy" containerID="cri-o://e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92" gracePeriod=30
Jul 28 16:16:54.594744 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:54.594716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:16:55.321299 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.321267    2580 generic.go:356] "Generic (PLEG): container finished" podID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerID="e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92" exitCode=2
Jul 28 16:16:55.321471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.321337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerDied","Data":"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"}
Jul 28 16:16:55.823448 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.823425    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:55.920197 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920073    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") pod \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") "
Jul 28 16:16:55.920379 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh4ql\" (UniqueName: \"kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql\") pod \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") "
Jul 28 16:16:55.920379 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert\") pod \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") "
Jul 28 16:16:55.920379 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920266    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location\") pod \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") "
Jul 28 16:16:55.920562 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920447    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config\") pod \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\" (UID: \"09d238cf-2ef9-4546-a3f7-537c27fea6f4\") "
Jul 28 16:16:55.920616 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09d238cf-2ef9-4546-a3f7-537c27fea6f4" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:16:55.920616 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09d238cf-2ef9-4546-a3f7-537c27fea6f4" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:16:55.920765 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09d238cf-2ef9-4546-a3f7-537c27fea6f4" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:16:55.921039 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.920944    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:16:55.921078 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.921047    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:16:55.921078 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.921062    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:16:55.922448 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.922419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql" (OuterVolumeSpecName: "kube-api-access-wh4ql") pod "09d238cf-2ef9-4546-a3f7-537c27fea6f4" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4"). InnerVolumeSpecName "kube-api-access-wh4ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:16:55.922549 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:55.922505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09d238cf-2ef9-4546-a3f7-537c27fea6f4" (UID: "09d238cf-2ef9-4546-a3f7-537c27fea6f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:16:56.022216 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.022178    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09d238cf-2ef9-4546-a3f7-537c27fea6f4-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:16:56.022216 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.022210    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4ql\" (UniqueName: \"kubernetes.io/projected/09d238cf-2ef9-4546-a3f7-537c27fea6f4-kube-api-access-wh4ql\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:16:56.327861 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.327825    2580 generic.go:356] "Generic (PLEG): container finished" podID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerID="a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4" exitCode=0
Jul 28 16:16:56.328013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.327888    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"
Jul 28 16:16:56.328013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.327910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerDied","Data":"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"}
Jul 28 16:16:56.328013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.327979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9" event={"ID":"09d238cf-2ef9-4546-a3f7-537c27fea6f4","Type":"ContainerDied","Data":"41dea8f6ec77970222973b7544019e0810ccac3b0583f1d5ce94c6e2db899368"}
Jul 28 16:16:56.328013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.328009    2580 scope.go:117] "RemoveContainer" containerID="e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"
Jul 28 16:16:56.336519 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.336499    2580 scope.go:117] "RemoveContainer" containerID="a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"
Jul 28 16:16:56.344220 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.344200    2580 scope.go:117] "RemoveContainer" containerID="57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"
Jul 28 16:16:56.347341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.347320    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:56.353169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353100    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6hdt9"]
Jul 28 16:16:56.353255 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353195    2580 scope.go:117] "RemoveContainer" containerID="e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"
Jul 28 16:16:56.353496 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:16:56.353478    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92\": container with ID starting with e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92 not found: ID does not exist" containerID="e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"
Jul 28 16:16:56.353545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353505    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92"} err="failed to get container status \"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92\": rpc error: code = NotFound desc = could not find container \"e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92\": container with ID starting with e66aae8acc79c46368eb09e253e0de5c7505aedcd296dd1677d6d53d228c4c92 not found: ID does not exist"
Jul 28 16:16:56.353545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353523    2580 scope.go:117] "RemoveContainer" containerID="a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"
Jul 28 16:16:56.353762 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:16:56.353743    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4\": container with ID starting with a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4 not found: ID does not exist" containerID="a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"
Jul 28 16:16:56.353834 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353774    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4"} err="failed to get container status \"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4\": rpc error: code = NotFound desc = could not find container \"a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4\": container with ID starting with a52357b6a43cad80e3d3bf0e522da477315b776690887b0e8422045303f9d3c4 not found: ID does not exist"
Jul 28 16:16:56.353834 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.353798    2580 scope.go:117] "RemoveContainer" containerID="57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"
Jul 28 16:16:56.354074 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:16:56.354057    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b\": container with ID starting with 57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b not found: ID does not exist" containerID="57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"
Jul 28 16:16:56.354113 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:56.354089    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b"} err="failed to get container status \"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b\": rpc error: code = NotFound desc = could not find container \"57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b\": container with ID starting with 57cc6291fd13c706102a725ea510e9c7ffffa505013f326fb716bf445f7ea61b not found: ID does not exist"
Jul 28 16:16:58.186361 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:58.186316    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" path="/var/lib/kubelet/pods/09d238cf-2ef9-4546-a3f7-537c27fea6f4/volumes"
Jul 28 16:16:59.727540 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.727507    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:16:59.858082 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858046    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:16:59.858426 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858414    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="storage-initializer"
Jul 28 16:16:59.858473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858428    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="storage-initializer"
Jul 28 16:16:59.858473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858438    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kube-rbac-proxy"
Jul 28 16:16:59.858473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858444    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kube-rbac-proxy"
Jul 28 16:16:59.858473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kserve-container"
Jul 28 16:16:59.858473 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858457    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kserve-container"
Jul 28 16:16:59.858626 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858518    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kserve-container"
Jul 28 16:16:59.858626 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.858526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d238cf-2ef9-4546-a3f7-537c27fea6f4" containerName="kube-rbac-proxy"
Jul 28 16:16:59.861914 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.861886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:16:59.865652 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.865628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:16:59.866015 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.865999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:16:59.866279 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.866262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:16:59.866377 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.866312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:16:59.867433 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.867411    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:16:59.867545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.867483    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 16:16:59.872459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.871986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 16:16:59.884612 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.884576    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:16:59.956171 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.956107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:16:59.956333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.956181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxpt\" (UniqueName: \"kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:16:59.956333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.956240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:16:59.956333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.956260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:16:59.956466 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:16:59.956328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.041326 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.041224    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:17:00.057359 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.057498 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.057498 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.057498 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.057670 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxpt\" (UniqueName: \"kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.057862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.057837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.058093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.058057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.058093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.058086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.060120 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.060098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.071004 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.070981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxpt\" (UniqueName: \"kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt\") pod \"isvc-sklearn-mcp-predictor-5996957568-r6tzq\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.172917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.172876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:00.390971 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:00.390939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:17:00.392325 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:17:00.392295    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1cadb7_8138_42fb_956b_3c9c2b7e6c04.slice/crio-1bc46b9473f6145f547eb4e404b79252e67a431dd62cf5c76d1c37e011eaccd7 WatchSource:0}: Error finding container 1bc46b9473f6145f547eb4e404b79252e67a431dd62cf5c76d1c37e011eaccd7: Status 404 returned error can't find the container with id 1bc46b9473f6145f547eb4e404b79252e67a431dd62cf5c76d1c37e011eaccd7
Jul 28 16:17:01.351624 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:01.351585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerStarted","Data":"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"}
Jul 28 16:17:01.352023 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:01.351650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerStarted","Data":"1bc46b9473f6145f547eb4e404b79252e67a431dd62cf5c76d1c37e011eaccd7"}
Jul 28 16:17:04.368918 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:04.368884    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerID="527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464" exitCode=0
Jul 28 16:17:04.369301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:04.368950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerDied","Data":"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"}
Jul 28 16:17:05.378369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:05.378326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerStarted","Data":"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"}
Jul 28 16:17:07.394217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:07.394180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerStarted","Data":"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"}
Jul 28 16:17:07.394606 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:07.394251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerStarted","Data":"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"}
Jul 28 16:17:07.394606 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:07.394360    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:07.394606 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:07.394423    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:07.420544 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:07.420482    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podStartSLOduration=1.843644488 podStartE2EDuration="8.420464211s" totalImagesPullingTime="2.782460985s" totalInitContainerRuntime="3.794358738s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:17:07.417881575 +0000 UTC m=+2641.786534538" watchObservedRunningTime="2026-07-28 16:17:07.420464211 +0000 UTC m=+2641.789117127"
Jul 28 16:17:08.398378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:08.398348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:14.408972 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:14.408944    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:17:34.410428 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:34.410389    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:17:40.174284 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:40.174221    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:17:44.414369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:17:44.414339    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:01.757482 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:01.757403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:18:14.415578 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:14.415549    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:19.877344 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:19.877309    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:18:19.877795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:19.877665    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" containerID="cri-o://aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c" gracePeriod=30
Jul 28 16:18:19.877795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:19.877671    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" containerID="cri-o://3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377" gracePeriod=30
Jul 28 16:18:19.877795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:19.877679    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-agent" containerID="cri-o://c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29" gracePeriod=30
Jul 28 16:18:19.899086 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:19.899053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:18:20.711994 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:20.711961    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerID="3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377" exitCode=2
Jul 28 16:18:20.711994 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:20.712003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerDied","Data":"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"}
Jul 28 16:18:22.723897 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:22.723862    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerID="aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c" exitCode=0
Jul 28 16:18:22.724317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:22.723939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerDied","Data":"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"}
Jul 28 16:18:24.404006 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:24.403958    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:24.409737 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:24.409711    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:18:25.010169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:25.010109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:18:25.237434 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:25.237392    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:18:29.403483 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:29.403439    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:34.403299 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:34.403259    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:34.403695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:34.403383    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:34.409809 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:34.409777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:18:39.403546 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:39.403497    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:44.403336 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:44.403288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:44.409794 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:44.409763    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.44:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:18:44.409920 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:44.409906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:49.403549 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:49.403506    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:18:50.071410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.071386    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:50.217646 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.217564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls\") pod \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") "
Jul 28 16:18:50.217795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.217651    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config\") pod \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") "
Jul 28 16:18:50.217795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.217694    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location\") pod \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") "
Jul 28 16:18:50.217795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.217740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert\") pod \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") "
Jul 28 16:18:50.217795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.217785    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqxpt\" (UniqueName: \"kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt\") pod \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\" (UID: \"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04\") "
Jul 28 16:18:50.218087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.218051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" (UID: "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:18:50.218087 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.218054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" (UID: "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:18:50.218297 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.218234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" (UID: "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:18:50.220168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.220119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt" (OuterVolumeSpecName: "kube-api-access-jqxpt") pod "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" (UID: "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04"). InnerVolumeSpecName "kube-api-access-jqxpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:18:50.220295 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.220262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" (UID: "bd1cadb7-8138-42fb-956b-3c9c2b7e6c04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:18:50.319587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.319379    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:18:50.319587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.319418    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:18:50.319587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.319437    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxpt\" (UniqueName: \"kubernetes.io/projected/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-api-access-jqxpt\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:18:50.319587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.319500    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:18:50.319587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.319540    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:18:50.854017 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.853981    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerID="c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29" exitCode=137
Jul 28 16:18:50.854516 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.854051    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"
Jul 28 16:18:50.854516 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.854052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerDied","Data":"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"}
Jul 28 16:18:50.854516 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.854226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq" event={"ID":"bd1cadb7-8138-42fb-956b-3c9c2b7e6c04","Type":"ContainerDied","Data":"1bc46b9473f6145f547eb4e404b79252e67a431dd62cf5c76d1c37e011eaccd7"}
Jul 28 16:18:50.854516 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.854257    2580 scope.go:117] "RemoveContainer" containerID="3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"
Jul 28 16:18:50.863504 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.863488    2580 scope.go:117] "RemoveContainer" containerID="c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"
Jul 28 16:18:50.871050 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.871030    2580 scope.go:117] "RemoveContainer" containerID="aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"
Jul 28 16:18:50.877431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.877407    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:18:50.879182 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.878984    2580 scope.go:117] "RemoveContainer" containerID="527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"
Jul 28 16:18:50.881690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.881667    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-r6tzq"]
Jul 28 16:18:50.887420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.887300    2580 scope.go:117] "RemoveContainer" containerID="3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"
Jul 28 16:18:50.887617 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:18:50.887596    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377\": container with ID starting with 3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377 not found: ID does not exist" containerID="3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"
Jul 28 16:18:50.887687 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.887633    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377"} err="failed to get container status \"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377\": rpc error: code = NotFound desc = could not find container \"3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377\": container with ID starting with 3c3a1a7cd03fe7d0d137e95d597d9e6c345bfa6e29bcba7283ed41cc67655377 not found: ID does not exist"
Jul 28 16:18:50.887687 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.887652    2580 scope.go:117] "RemoveContainer" containerID="c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"
Jul 28 16:18:50.887888 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:18:50.887870    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29\": container with ID starting with c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29 not found: ID does not exist" containerID="c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"
Jul 28 16:18:50.887960 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.887898    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29"} err="failed to get container status \"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29\": rpc error: code = NotFound desc = could not find container \"c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29\": container with ID starting with c72a8180eb2b91d7d258d801c9d84f465e139a7cd3275d9f889bf85b34d1dd29 not found: ID does not exist"
Jul 28 16:18:50.887960 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.887920    2580 scope.go:117] "RemoveContainer" containerID="aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"
Jul 28 16:18:50.888196 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:18:50.888181    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c\": container with ID starting with aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c not found: ID does not exist" containerID="aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"
Jul 28 16:18:50.888253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.888198    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c"} err="failed to get container status \"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c\": rpc error: code = NotFound desc = could not find container \"aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c\": container with ID starting with aefcd7d80cb00e57d50fa3415cf3d0aad7a150f8d6638f30c34f630825a1c82c not found: ID does not exist"
Jul 28 16:18:50.888253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.888234    2580 scope.go:117] "RemoveContainer" containerID="527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"
Jul 28 16:18:50.888445 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:18:50.888427    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464\": container with ID starting with 527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464 not found: ID does not exist" containerID="527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"
Jul 28 16:18:50.888483 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:50.888447    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464"} err="failed to get container status \"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464\": rpc error: code = NotFound desc = could not find container \"527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464\": container with ID starting with 527e09fae4b1b3b2f6e308879030c80dc0141c9197656ba613552166a758e464 not found: ID does not exist"
Jul 28 16:18:52.185938 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:18:52.185906    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" path="/var/lib/kubelet/pods/bd1cadb7-8138-42fb-956b-3c9c2b7e6c04/volumes"
Jul 28 16:19:36.822470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:36.822378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:19:41.912221 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.912184    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:19:41.952961 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.952916    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:19:41.953363 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953345    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953366    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953391    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-agent"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953401    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-agent"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953419    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953427    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953448    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="storage-initializer"
Jul 28 16:19:41.953463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953457    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="storage-initializer"
Jul 28 16:19:41.953822 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953538    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-container"
Jul 28 16:19:41.953822 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953554    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kserve-agent"
Jul 28 16:19:41.953822 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.953563    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd1cadb7-8138-42fb-956b-3c9c2b7e6c04" containerName="kube-rbac-proxy"
Jul 28 16:19:41.956625 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.956602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:41.959431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.959398    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:19:41.959431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.959417    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:19:41.959758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.959740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:19:41.960802 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.960772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 16:19:41.960915 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.960832    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:19:41.960915 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.960869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:19:41.961031 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.960953    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:19:41.969410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.969389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:19:41.990220 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:41.990188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:19:42.073260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.073225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.073260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.073263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.073464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.073336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.073464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.073365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rh6l\" (UniqueName: \"kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.073464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.073385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.174853 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.174770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.174853 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.174808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175018 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.174858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175018 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.174899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rh6l\" (UniqueName: \"kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175018 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.174931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175428 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.175406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175599 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.175581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.175665 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.175648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.177489 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.177471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.185165 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.185143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rh6l\" (UniqueName: \"kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l\") pod \"isvc-paddle-runtime-predictor-58b99fff76-cnhfv\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.267669 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.267628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:19:42.407192 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:42.407170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:19:42.410079 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:19:42.410054    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba8dbd0_ef68_4ad7_8518_e13004c65470.slice/crio-8699598c2bf6ea6400b04b9183a1cb3b7303a01ef7ce7f23062f98d4361af652 WatchSource:0}: Error finding container 8699598c2bf6ea6400b04b9183a1cb3b7303a01ef7ce7f23062f98d4361af652: Status 404 returned error can't find the container with id 8699598c2bf6ea6400b04b9183a1cb3b7303a01ef7ce7f23062f98d4361af652
Jul 28 16:19:43.079761 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:43.079720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerStarted","Data":"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"}
Jul 28 16:19:43.080188 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:43.079824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerStarted","Data":"8699598c2bf6ea6400b04b9183a1cb3b7303a01ef7ce7f23062f98d4361af652"}
Jul 28 16:19:48.104760 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:48.104729    2580 generic.go:356] "Generic (PLEG): container finished" podID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerID="228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca" exitCode=0
Jul 28 16:19:48.105169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:48.104809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerDied","Data":"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"}
Jul 28 16:19:48.106079 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:19:48.106064    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:20:00.169182 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:00.169119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerStarted","Data":"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"}
Jul 28 16:20:00.169611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:00.169225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerStarted","Data":"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"}
Jul 28 16:20:00.169611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:00.169440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:00.192504 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:00.192458    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podStartSLOduration=2.970102133 podStartE2EDuration="19.19244308s" totalImagesPullingTime="11.28621915s" totalInitContainerRuntime="4.936121797s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:00.19126935 +0000 UTC m=+2814.559922265" watchObservedRunningTime="2026-07-28 16:20:00.19244308 +0000 UTC m=+2814.561096065"
Jul 28 16:20:01.173673 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:01.173640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:01.175068 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:01.175040    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:02.178396 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:02.178355    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:07.183379 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:07.183348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:07.184032 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:07.183994    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:11.951890 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:11.951839    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:20:17.183982 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:17.183933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:27.183988 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:27.183942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:37.183919 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:37.183877    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:20:47.184923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:47.184892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:53.645582 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:53.645549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:20:53.646094 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:53.645812    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container" containerID="cri-o://aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9" gracePeriod=30
Jul 28 16:20:53.646094 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:53.645851    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kube-rbac-proxy" containerID="cri-o://661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd" gracePeriod=30
Jul 28 16:20:53.670789 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:53.670734    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:20:54.406289 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:54.406257    2580 generic.go:356] "Generic (PLEG): container finished" podID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerID="661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd" exitCode=2
Jul 28 16:20:54.406463 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:54.406307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerDied","Data":"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"}
Jul 28 16:20:56.390810 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.390783    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:56.417006 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.416971    2580 generic.go:356] "Generic (PLEG): container finished" podID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerID="aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9" exitCode=0
Jul 28 16:20:56.417218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.417033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"
Jul 28 16:20:56.417218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.417048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerDied","Data":"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"}
Jul 28 16:20:56.417218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.417139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv" event={"ID":"eba8dbd0-ef68-4ad7-8518-e13004c65470","Type":"ContainerDied","Data":"8699598c2bf6ea6400b04b9183a1cb3b7303a01ef7ce7f23062f98d4361af652"}
Jul 28 16:20:56.417218 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.417170    2580 scope.go:117] "RemoveContainer" containerID="661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"
Jul 28 16:20:56.426180 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.426160    2580 scope.go:117] "RemoveContainer" containerID="aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"
Jul 28 16:20:56.434232 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.434214    2580 scope.go:117] "RemoveContainer" containerID="228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"
Jul 28 16:20:56.442302 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.442282    2580 scope.go:117] "RemoveContainer" containerID="661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"
Jul 28 16:20:56.442576 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:20:56.442556    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd\": container with ID starting with 661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd not found: ID does not exist" containerID="661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"
Jul 28 16:20:56.442619 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.442589    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd"} err="failed to get container status \"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd\": rpc error: code = NotFound desc = could not find container \"661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd\": container with ID starting with 661b9742a28cb89b9f3bf29a1bf8b32390549fbce42b9835ada49caa02b7e8fd not found: ID does not exist"
Jul 28 16:20:56.442660 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.442619    2580 scope.go:117] "RemoveContainer" containerID="aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"
Jul 28 16:20:56.442874 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:20:56.442857    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9\": container with ID starting with aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9 not found: ID does not exist" containerID="aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"
Jul 28 16:20:56.442917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.442882    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9"} err="failed to get container status \"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9\": rpc error: code = NotFound desc = could not find container \"aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9\": container with ID starting with aa6b2415f5a9f8f4e7928f19b774f30271e6af3399cef9cfbb0f8853f2f41ae9 not found: ID does not exist"
Jul 28 16:20:56.442917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.442897    2580 scope.go:117] "RemoveContainer" containerID="228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"
Jul 28 16:20:56.443215 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:20:56.443190    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca\": container with ID starting with 228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca not found: ID does not exist" containerID="228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"
Jul 28 16:20:56.443283 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.443223    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca"} err="failed to get container status \"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca\": rpc error: code = NotFound desc = could not find container \"228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca\": container with ID starting with 228e663e4e73f2fd823334e749c5cbc5b37a0c08f979a21efa6f74ede8e1a1ca not found: ID does not exist"
Jul 28 16:20:56.449500 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.449481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location\") pod \"eba8dbd0-ef68-4ad7-8518-e13004c65470\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") "
Jul 28 16:20:56.449567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.449513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls\") pod \"eba8dbd0-ef68-4ad7-8518-e13004c65470\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") "
Jul 28 16:20:56.449610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.449602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert\") pod \"eba8dbd0-ef68-4ad7-8518-e13004c65470\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") "
Jul 28 16:20:56.449656 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.449631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config\") pod \"eba8dbd0-ef68-4ad7-8518-e13004c65470\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") "
Jul 28 16:20:56.449702 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.449664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rh6l\" (UniqueName: \"kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l\") pod \"eba8dbd0-ef68-4ad7-8518-e13004c65470\" (UID: \"eba8dbd0-ef68-4ad7-8518-e13004c65470\") "
Jul 28 16:20:56.450027 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.450002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eba8dbd0-ef68-4ad7-8518-e13004c65470" (UID: "eba8dbd0-ef68-4ad7-8518-e13004c65470"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:20:56.450096 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.450017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eba8dbd0-ef68-4ad7-8518-e13004c65470" (UID: "eba8dbd0-ef68-4ad7-8518-e13004c65470"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:20:56.451681 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.451663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eba8dbd0-ef68-4ad7-8518-e13004c65470" (UID: "eba8dbd0-ef68-4ad7-8518-e13004c65470"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:20:56.452115 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.452101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l" (OuterVolumeSpecName: "kube-api-access-8rh6l") pod "eba8dbd0-ef68-4ad7-8518-e13004c65470" (UID: "eba8dbd0-ef68-4ad7-8518-e13004c65470"). InnerVolumeSpecName "kube-api-access-8rh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:20:56.460394 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.460374    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eba8dbd0-ef68-4ad7-8518-e13004c65470" (UID: "eba8dbd0-ef68-4ad7-8518-e13004c65470"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:20:56.550941 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.550890    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eba8dbd0-ef68-4ad7-8518-e13004c65470-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:20:56.550941 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.550938    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eba8dbd0-ef68-4ad7-8518-e13004c65470-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:20:56.550941 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.550949    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:20:56.551204 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.550959    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:20:56.551204 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.550968    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rh6l\" (UniqueName: \"kubernetes.io/projected/eba8dbd0-ef68-4ad7-8518-e13004c65470-kube-api-access-8rh6l\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:20:56.740283 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.740252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:20:56.744010 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:56.743986    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-cnhfv"]
Jul 28 16:20:58.186166 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.186110    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" path="/var/lib/kubelet/pods/eba8dbd0-ef68-4ad7-8518-e13004c65470/volumes"
Jul 28 16:20:58.788583 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.788533    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:20:58.821627 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821597    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:20:58.821946 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821934    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container"
Jul 28 16:20:58.822000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821947    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container"
Jul 28 16:20:58.822000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821968    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kube-rbac-proxy"
Jul 28 16:20:58.822000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821974    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kube-rbac-proxy"
Jul 28 16:20:58.822000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821985    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="storage-initializer"
Jul 28 16:20:58.822000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.821991    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="storage-initializer"
Jul 28 16:20:58.822201 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.822044    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kserve-container"
Jul 28 16:20:58.822201 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.822054    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba8dbd0-ef68-4ad7-8518-e13004c65470" containerName="kube-rbac-proxy"
Jul 28 16:20:58.824836 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.824820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:58.827664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.827634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:20:58.828850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828832    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:20:58.828964 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:20:58.828964 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:20:58.828964 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828929    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 16:20:58.828964 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828863    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 16:20:58.829221 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.828983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:20:58.837849 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.837825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:20:58.859557 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.859531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:20:58.972078 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.972043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:58.972259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.972120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:58.972259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.972176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpcm\" (UniqueName: \"kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:58.972259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.972196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:58.972259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:58.972220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpcm\" (UniqueName: \"kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073326    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073673 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073755 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.073989 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.073963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.074080 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.074066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.075909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.075892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.083075 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.083054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpcm\" (UniqueName: \"kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.135621 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.135584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:20:59.272896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.272869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:20:59.275279 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:20:59.275249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e1a883_349a_40bb_9db8_52df9ee6da45.slice/crio-110625d75939682f4098d93ff9b7d72077d362288a91e3df0e704dabd3e02eae WatchSource:0}: Error finding container 110625d75939682f4098d93ff9b7d72077d362288a91e3df0e704dabd3e02eae: Status 404 returned error can't find the container with id 110625d75939682f4098d93ff9b7d72077d362288a91e3df0e704dabd3e02eae
Jul 28 16:20:59.433160 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.433110    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerStarted","Data":"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"}
Jul 28 16:20:59.433335 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:20:59.433185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerStarted","Data":"110625d75939682f4098d93ff9b7d72077d362288a91e3df0e704dabd3e02eae"}
Jul 28 16:21:04.458517 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:04.458482    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerID="c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4" exitCode=0
Jul 28 16:21:04.458925 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:04.458552    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerDied","Data":"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"}
Jul 28 16:21:05.465831 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:05.465794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerStarted","Data":"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"}
Jul 28 16:21:05.466317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:05.465864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerStarted","Data":"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"}
Jul 28 16:21:05.466317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:05.466090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:21:05.489919 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:05.489865    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podStartSLOduration=3.214249424 podStartE2EDuration="7.489849483s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.275600059s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:05.488761414 +0000 UTC m=+2879.857414332" watchObservedRunningTime="2026-07-28 16:21:05.489849483 +0000 UTC m=+2879.858502399"
Jul 28 16:21:06.471934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:06.471904    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:21:06.472976 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:06.472947    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:07.475832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:07.475789    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:12.480555 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:12.480525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:21:12.481279 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:12.481243    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:22.481693 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:22.481646    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:32.481742 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:32.481701    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:42.481632 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:42.481588    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:21:52.481813 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:21:52.481780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:22:00.763234 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:00.763198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:22:00.763640 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:00.763493    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kube-rbac-proxy" containerID="cri-o://d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5" gracePeriod=30
Jul 28 16:22:00.763640 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:00.763493    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" containerID="cri-o://6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11" gracePeriod=30
Jul 28 16:22:00.776372 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:00.776349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:01.713116 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:01.713074    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerID="d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5" exitCode=2
Jul 28 16:22:01.713301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:01.713154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerDied","Data":"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"}
Jul 28 16:22:02.476472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:02.476430    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 28 16:22:02.481907 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:02.481869    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:22:03.516052 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.516028    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:22:03.550605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.550525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location\") pod \"f0e1a883-349a-40bb-9db8-52df9ee6da45\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") "
Jul 28 16:22:03.550605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.550590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrpcm\" (UniqueName: \"kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm\") pod \"f0e1a883-349a-40bb-9db8-52df9ee6da45\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") "
Jul 28 16:22:03.550823 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.550621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls\") pod \"f0e1a883-349a-40bb-9db8-52df9ee6da45\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") "
Jul 28 16:22:03.550823 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.550641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert\") pod \"f0e1a883-349a-40bb-9db8-52df9ee6da45\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") "
Jul 28 16:22:03.550823 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.550724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config\") pod \"f0e1a883-349a-40bb-9db8-52df9ee6da45\" (UID: \"f0e1a883-349a-40bb-9db8-52df9ee6da45\") "
Jul 28 16:22:03.551109 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.551061    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0e1a883-349a-40bb-9db8-52df9ee6da45" (UID: "f0e1a883-349a-40bb-9db8-52df9ee6da45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:03.551253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.551109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0e1a883-349a-40bb-9db8-52df9ee6da45" (UID: "f0e1a883-349a-40bb-9db8-52df9ee6da45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:03.553402 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.553366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0e1a883-349a-40bb-9db8-52df9ee6da45" (UID: "f0e1a883-349a-40bb-9db8-52df9ee6da45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:03.553498 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.553380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm" (OuterVolumeSpecName: "kube-api-access-qrpcm") pod "f0e1a883-349a-40bb-9db8-52df9ee6da45" (UID: "f0e1a883-349a-40bb-9db8-52df9ee6da45"). InnerVolumeSpecName "kube-api-access-qrpcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:03.560513 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.560489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0e1a883-349a-40bb-9db8-52df9ee6da45" (UID: "f0e1a883-349a-40bb-9db8-52df9ee6da45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:22:03.651688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.651657    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpcm\" (UniqueName: \"kubernetes.io/projected/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-api-access-qrpcm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:22:03.651688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.651682    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e1a883-349a-40bb-9db8-52df9ee6da45-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:22:03.651688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.651694    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:22:03.651921 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.651703    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e1a883-349a-40bb-9db8-52df9ee6da45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:22:03.651921 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.651713    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e1a883-349a-40bb-9db8-52df9ee6da45-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:22:03.724423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.724387    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerID="6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11" exitCode=0
Jul 28 16:22:03.724423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.724424    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerDied","Data":"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"}
Jul 28 16:22:03.724630 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.724448    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"
Jul 28 16:22:03.724630 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.724474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh" event={"ID":"f0e1a883-349a-40bb-9db8-52df9ee6da45","Type":"ContainerDied","Data":"110625d75939682f4098d93ff9b7d72077d362288a91e3df0e704dabd3e02eae"}
Jul 28 16:22:03.724630 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.724500    2580 scope.go:117] "RemoveContainer" containerID="d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"
Jul 28 16:22:03.733737 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.733720    2580 scope.go:117] "RemoveContainer" containerID="6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"
Jul 28 16:22:03.743659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.743621    2580 scope.go:117] "RemoveContainer" containerID="c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"
Jul 28 16:22:03.748999 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.748974    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:22:03.751757 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.751728    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-8ppwh"]
Jul 28 16:22:03.753454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.753435    2580 scope.go:117] "RemoveContainer" containerID="d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"
Jul 28 16:22:03.753740 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:22:03.753720    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5\": container with ID starting with d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5 not found: ID does not exist" containerID="d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"
Jul 28 16:22:03.753796 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.753749    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5"} err="failed to get container status \"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5\": rpc error: code = NotFound desc = could not find container \"d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5\": container with ID starting with d45904d63b34be67d3b9e4c7ee89bfc18762329dcec15f5de0241acaac2b15b5 not found: ID does not exist"
Jul 28 16:22:03.753796 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.753768    2580 scope.go:117] "RemoveContainer" containerID="6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"
Jul 28 16:22:03.754033 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:22:03.754015    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11\": container with ID starting with 6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11 not found: ID does not exist" containerID="6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"
Jul 28 16:22:03.754083 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.754037    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11"} err="failed to get container status \"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11\": rpc error: code = NotFound desc = could not find container \"6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11\": container with ID starting with 6943e7615e58125fa2422d8604084c5d835ce5caf130af2d3dc277c44c237d11 not found: ID does not exist"
Jul 28 16:22:03.754083 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.754049    2580 scope.go:117] "RemoveContainer" containerID="c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"
Jul 28 16:22:03.754334 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:22:03.754319    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4\": container with ID starting with c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4 not found: ID does not exist" containerID="c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"
Jul 28 16:22:03.754378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:03.754338    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4"} err="failed to get container status \"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4\": rpc error: code = NotFound desc = could not find container \"c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4\": container with ID starting with c813d26dc7282cb08ae07025e67d3d5ebc12f93cb99f64254477885e4e108fc4 not found: ID does not exist"
Jul 28 16:22:04.185576 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:04.185540    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" path="/var/lib/kubelet/pods/f0e1a883-349a-40bb-9db8-52df9ee6da45/volumes"
Jul 28 16:22:05.894520 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.894489    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:05.941053 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:22:05.941424 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941408    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="storage-initializer"
Jul 28 16:22:05.941470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941426    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="storage-initializer"
Jul 28 16:22:05.941470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941443    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kube-rbac-proxy"
Jul 28 16:22:05.941470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941449    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kube-rbac-proxy"
Jul 28 16:22:05.941470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941467    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container"
Jul 28 16:22:05.941590 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941473    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container"
Jul 28 16:22:05.941590 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941527    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kube-rbac-proxy"
Jul 28 16:22:05.941590 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.941536    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e1a883-349a-40bb-9db8-52df9ee6da45" containerName="kserve-container"
Jul 28 16:22:05.946120 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.946104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:05.949044 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.949022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:22:05.949044 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.949039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:22:05.949245 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.949070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 16:22:05.949245 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.949049    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:22:05.950370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.950340    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 16:22:05.950478 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.950401    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:22:05.950478 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.950444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:22:05.955331 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.955309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:22:05.970512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.970379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:05.970512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.970435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:05.970512 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.970496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:05.970774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.970525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq29l\" (UniqueName: \"kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:05.970774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:05.970552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071026 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.070994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071231 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.071036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071231 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.071071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071231 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.071110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq29l\" (UniqueName: \"kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071231 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.071157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.071471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.071453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.074066 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.074043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 16:22:06.074066 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.074064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 16:22:06.074240 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.074050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:22:06.081778 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.081749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.081885 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.081796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.083617 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.083595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.083970 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.083954    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:22:06.094869 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.094851    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:22:06.099672 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.099623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:22:06.105085 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.105067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq29l\" (UniqueName: \"kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l\") pod \"isvc-pmml-predictor-8674795f59-zrssw\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.260164 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.260065    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:22:06.267839 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.267810    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:06.406342 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.406312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:22:06.408239 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:22:06.408205    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50228b64_9ce0_46d7_9d3f_dd16fcf00791.slice/crio-c524c03e750e5d2a0e0ae89b1147d03c93ae4b816d1a9ee3836d7240b8ccaeac WatchSource:0}: Error finding container c524c03e750e5d2a0e0ae89b1147d03c93ae4b816d1a9ee3836d7240b8ccaeac: Status 404 returned error can't find the container with id c524c03e750e5d2a0e0ae89b1147d03c93ae4b816d1a9ee3836d7240b8ccaeac
Jul 28 16:22:06.413189 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.413164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:22:06.739741 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.739709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerStarted","Data":"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"}
Jul 28 16:22:06.739940 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:06.739774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerStarted","Data":"c524c03e750e5d2a0e0ae89b1147d03c93ae4b816d1a9ee3836d7240b8ccaeac"}
Jul 28 16:22:10.757543 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:10.757511    2580 generic.go:356] "Generic (PLEG): container finished" podID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerID="788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb" exitCode=0
Jul 28 16:22:10.757919 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:10.757573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerDied","Data":"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"}
Jul 28 16:22:18.803796 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:18.803759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerStarted","Data":"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"}
Jul 28 16:22:18.804328 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:18.803839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerStarted","Data":"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"}
Jul 28 16:22:18.804328 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:18.804051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:18.825431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:18.825371    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podStartSLOduration=3.049800713 podStartE2EDuration="13.825355659s" totalImagesPullingTime="7.043443748s" totalInitContainerRuntime="3.732111198s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:18.824158376 +0000 UTC m=+2953.192811297" watchObservedRunningTime="2026-07-28 16:22:18.825355659 +0000 UTC m=+2953.194008574"
Jul 28 16:22:19.807959 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:19.807923    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:19.809320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:19.809291    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:22:20.812260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:20.812211    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:22:25.817276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:25.817239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:22:25.817758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:25.817732    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:22:35.818583 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:35.818540    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:22:45.818589 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:45.818550    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:22:55.818523 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:22:55.818483    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:23:05.818695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:05.818654    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:23:14.084033 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:14.083992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:15.818708 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:15.818670    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:23:25.818486 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:25.818440    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:23:35.818692 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:35.818651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:23:45.818300 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:45.818265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:23:57.107488 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.107443    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:23:57.107980 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.107778    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container" containerID="cri-o://01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0" gracePeriod=30
Jul 28 16:23:57.107980 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.107850    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kube-rbac-proxy" containerID="cri-o://9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059" gracePeriod=30
Jul 28 16:23:57.150789 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.150756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:23:57.233940 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.233911    2580 generic.go:356] "Generic (PLEG): container finished" podID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerID="9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059" exitCode=2
Jul 28 16:23:57.234098 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:23:57.233975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerDied","Data":"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"}
Jul 28 16:24:00.813047 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:00.812984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 16:24:00.960999 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:00.960972    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:24:01.098189 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert\") pod \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") "
Jul 28 16:24:01.098403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq29l\" (UniqueName: \"kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l\") pod \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") "
Jul 28 16:24:01.098403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config\") pod \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") "
Jul 28 16:24:01.098403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls\") pod \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") "
Jul 28 16:24:01.098403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location\") pod \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\" (UID: \"50228b64-9ce0-46d7-9d3f-dd16fcf00791\") "
Jul 28 16:24:01.098635 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50228b64-9ce0-46d7-9d3f-dd16fcf00791" (UID: "50228b64-9ce0-46d7-9d3f-dd16fcf00791"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:24:01.098730 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50228b64-9ce0-46d7-9d3f-dd16fcf00791" (UID: "50228b64-9ce0-46d7-9d3f-dd16fcf00791"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:24:01.098730 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.098715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50228b64-9ce0-46d7-9d3f-dd16fcf00791" (UID: "50228b64-9ce0-46d7-9d3f-dd16fcf00791"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:24:01.100575 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.100551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l" (OuterVolumeSpecName: "kube-api-access-jq29l") pod "50228b64-9ce0-46d7-9d3f-dd16fcf00791" (UID: "50228b64-9ce0-46d7-9d3f-dd16fcf00791"). InnerVolumeSpecName "kube-api-access-jq29l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:24:01.100729 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.100576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50228b64-9ce0-46d7-9d3f-dd16fcf00791" (UID: "50228b64-9ce0-46d7-9d3f-dd16fcf00791"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:24:01.199515 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.199478    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:24:01.199515 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.199505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq29l\" (UniqueName: \"kubernetes.io/projected/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-api-access-jq29l\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:24:01.199515 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.199516    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:24:01.199764 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.199527    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50228b64-9ce0-46d7-9d3f-dd16fcf00791-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:24:01.199764 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.199538    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50228b64-9ce0-46d7-9d3f-dd16fcf00791-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:24:01.252997 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.252960    2580 generic.go:356] "Generic (PLEG): container finished" podID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerID="01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0" exitCode=0
Jul 28 16:24:01.253206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.253008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerDied","Data":"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"}
Jul 28 16:24:01.253206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.253033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"
Jul 28 16:24:01.253206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.253076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw" event={"ID":"50228b64-9ce0-46d7-9d3f-dd16fcf00791","Type":"ContainerDied","Data":"c524c03e750e5d2a0e0ae89b1147d03c93ae4b816d1a9ee3836d7240b8ccaeac"}
Jul 28 16:24:01.253206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.253107    2580 scope.go:117] "RemoveContainer" containerID="9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"
Jul 28 16:24:01.262575 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.262554    2580 scope.go:117] "RemoveContainer" containerID="01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"
Jul 28 16:24:01.271001 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.270984    2580 scope.go:117] "RemoveContainer" containerID="788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"
Jul 28 16:24:01.281020 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.280995    2580 scope.go:117] "RemoveContainer" containerID="9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"
Jul 28 16:24:01.281436 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:24:01.281356    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059\": container with ID starting with 9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059 not found: ID does not exist" containerID="9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"
Jul 28 16:24:01.281436 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.281395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059"} err="failed to get container status \"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059\": rpc error: code = NotFound desc = could not find container \"9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059\": container with ID starting with 9656724d2cb9d7b1a5640bbf5ed986ae9e4c53aaec09f171bce8abf56ffc3059 not found: ID does not exist"
Jul 28 16:24:01.281436 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.281420    2580 scope.go:117] "RemoveContainer" containerID="01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"
Jul 28 16:24:01.281710 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:24:01.281692    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0\": container with ID starting with 01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0 not found: ID does not exist" containerID="01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"
Jul 28 16:24:01.281815 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.281719    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0"} err="failed to get container status \"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0\": rpc error: code = NotFound desc = could not find container \"01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0\": container with ID starting with 01cc5b72e0900a2f07d9f4a303c37785697d5d2b2cb5bbe3b49b634cac7260a0 not found: ID does not exist"
Jul 28 16:24:01.281815 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.281740    2580 scope.go:117] "RemoveContainer" containerID="788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"
Jul 28 16:24:01.282019 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:24:01.282002    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb\": container with ID starting with 788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb not found: ID does not exist" containerID="788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"
Jul 28 16:24:01.282059 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.282025    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb"} err="failed to get container status \"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb\": rpc error: code = NotFound desc = could not find container \"788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb\": container with ID starting with 788ed8c84e2a2ffcb407b35fa755766d66a5e92d6cf7722674708a022d5c5edb not found: ID does not exist"
Jul 28 16:24:01.282855 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.282837    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:24:01.296975 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:01.296945    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-zrssw"]
Jul 28 16:24:02.186187 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.186158    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" path="/var/lib/kubelet/pods/50228b64-9ce0-46d7-9d3f-dd16fcf00791/volumes"
Jul 28 16:24:02.208579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.208542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:24:02.255380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:24:02.255925 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255910    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container"
Jul 28 16:24:02.255974 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255929    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container"
Jul 28 16:24:02.255974 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255954    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="storage-initializer"
Jul 28 16:24:02.255974 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255963    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="storage-initializer"
Jul 28 16:24:02.256069 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255973    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kube-rbac-proxy"
Jul 28 16:24:02.256069 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.255982    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kube-rbac-proxy"
Jul 28 16:24:02.256147 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.256070    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kube-rbac-proxy"
Jul 28 16:24:02.256147 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.256081    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50228b64-9ce0-46d7-9d3f-dd16fcf00791" containerName="kserve-container"
Jul 28 16:24:02.260736 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.260712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.264017 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.263995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:24:02.264160 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.264032    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:24:02.264297 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.264284    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:24:02.264625 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.264605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 16:24:02.265381 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.265367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:24:02.265475 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.265461    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:24:02.265545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.265533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:24:02.273459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.273437    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:24:02.309155 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.309094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvp5h\" (UniqueName: \"kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.309383 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.309184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.309383 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.309222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.309383 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.309260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.309383 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.309284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410348 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvp5h\" (UniqueName: \"kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410348 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410618 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410618 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410618 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.410618 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:24:02.410571    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 16:24:02.410796 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:24:02.410674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls podName:cce1ff07-bb57-47be-ba8a-dbf358472824 nodeName:}" failed. No retries permitted until 2026-07-28 16:24:02.91065261 +0000 UTC m=+3057.279305504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-lhgmk" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 16:24:02.410845 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.410789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.411070 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.411054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.411110 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.411099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.420007 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.419979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvp5h\" (UniqueName: \"kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.486717 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.486643    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:24:02.915325 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.915272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:02.917811 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:02.917781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-lhgmk\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:03.171575 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:03.171491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:03.315884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:03.315855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:24:03.318346 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:24:03.318322    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce1ff07_bb57_47be_ba8a_dbf358472824.slice/crio-b6b35c43e92b02b58a4b7c347ad0e4654512e0e0207c7428625390447bd6adcb WatchSource:0}: Error finding container b6b35c43e92b02b58a4b7c347ad0e4654512e0e0207c7428625390447bd6adcb: Status 404 returned error can't find the container with id b6b35c43e92b02b58a4b7c347ad0e4654512e0e0207c7428625390447bd6adcb
Jul 28 16:24:04.272439 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:04.272400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerStarted","Data":"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"}
Jul 28 16:24:04.272629 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:04.272476    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerStarted","Data":"b6b35c43e92b02b58a4b7c347ad0e4654512e0e0207c7428625390447bd6adcb"}
Jul 28 16:24:07.288266 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:07.288232    2580 generic.go:356] "Generic (PLEG): container finished" podID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerID="76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0" exitCode=0
Jul 28 16:24:07.288669 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:07.288279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerDied","Data":"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"}
Jul 28 16:24:08.295108 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:08.295075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerStarted","Data":"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"}
Jul 28 16:24:08.295501 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:08.295158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerStarted","Data":"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"}
Jul 28 16:24:08.295501 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:08.295347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:08.333021 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:08.332966    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podStartSLOduration=2.616646879 podStartE2EDuration="6.332950154s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.716303275s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:24:08.331211759 +0000 UTC m=+3062.699864676" watchObservedRunningTime="2026-07-28 16:24:08.332950154 +0000 UTC m=+3062.701603122"
Jul 28 16:24:09.299877 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:09.299847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:09.301007 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:09.300977    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:10.304030 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:10.303985    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:15.308464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:15.308433    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:24:15.309169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:15.309117    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:25.309291 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:25.309246    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:35.309568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:35.309526    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:45.309403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:45.309358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:24:55.309376 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:24:55.309332    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:25:05.309887 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:05.309846    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:25:15.309154 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:15.309101    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:25:25.309271 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:25.309227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:25:35.310358 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:35.310276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:25:43.474456 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.474418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:25:43.474881 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.474672    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" containerID="cri-o://902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a" gracePeriod=30
Jul 28 16:25:43.474881 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.474718    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kube-rbac-proxy" containerID="cri-o://0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495" gracePeriod=30
Jul 28 16:25:43.530245 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.530207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:25:43.722870 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.722832    2580 generic.go:356] "Generic (PLEG): container finished" podID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerID="0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495" exitCode=2
Jul 28 16:25:43.723035 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:43.722904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerDied","Data":"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"}
Jul 28 16:25:45.304695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:45.304655    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 28 16:25:45.310081 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:45.310048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:25:47.326601 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.326569    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:25:47.424028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.423942    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location\") pod \"cce1ff07-bb57-47be-ba8a-dbf358472824\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") "
Jul 28 16:25:47.424028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424004    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") pod \"cce1ff07-bb57-47be-ba8a-dbf358472824\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") "
Jul 28 16:25:47.424028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert\") pod \"cce1ff07-bb57-47be-ba8a-dbf358472824\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") "
Jul 28 16:25:47.424349 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvp5h\" (UniqueName: \"kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h\") pod \"cce1ff07-bb57-47be-ba8a-dbf358472824\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") "
Jul 28 16:25:47.424349 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config\") pod \"cce1ff07-bb57-47be-ba8a-dbf358472824\" (UID: \"cce1ff07-bb57-47be-ba8a-dbf358472824\") "
Jul 28 16:25:47.424454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cce1ff07-bb57-47be-ba8a-dbf358472824" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:25:47.424454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424420    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cce1ff07-bb57-47be-ba8a-dbf358472824" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:25:47.424536 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424477    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cce1ff07-bb57-47be-ba8a-dbf358472824-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:25:47.424884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.424802    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cce1ff07-bb57-47be-ba8a-dbf358472824" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:25:47.426409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.426380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h" (OuterVolumeSpecName: "kube-api-access-dvp5h") pod "cce1ff07-bb57-47be-ba8a-dbf358472824" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824"). InnerVolumeSpecName "kube-api-access-dvp5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:25:47.426650 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.426634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cce1ff07-bb57-47be-ba8a-dbf358472824" (UID: "cce1ff07-bb57-47be-ba8a-dbf358472824"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:25:47.525238 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.525198    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:25:47.525238 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.525230    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce1ff07-bb57-47be-ba8a-dbf358472824-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:25:47.525238 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.525240    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cce1ff07-bb57-47be-ba8a-dbf358472824-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:25:47.525238 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.525249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvp5h\" (UniqueName: \"kubernetes.io/projected/cce1ff07-bb57-47be-ba8a-dbf358472824-kube-api-access-dvp5h\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:25:47.742150 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.742051    2580 generic.go:356] "Generic (PLEG): container finished" podID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerID="902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a" exitCode=0
Jul 28 16:25:47.742150 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.742104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerDied","Data":"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"}
Jul 28 16:25:47.742150 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.742108    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"
Jul 28 16:25:47.742406 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.742187    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk" event={"ID":"cce1ff07-bb57-47be-ba8a-dbf358472824","Type":"ContainerDied","Data":"b6b35c43e92b02b58a4b7c347ad0e4654512e0e0207c7428625390447bd6adcb"}
Jul 28 16:25:47.742406 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.742216    2580 scope.go:117] "RemoveContainer" containerID="0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"
Jul 28 16:25:47.751567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.751547    2580 scope.go:117] "RemoveContainer" containerID="902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"
Jul 28 16:25:47.760112 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.760094    2580 scope.go:117] "RemoveContainer" containerID="76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"
Jul 28 16:25:47.767262 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.767238    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:25:47.768464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.768449    2580 scope.go:117] "RemoveContainer" containerID="0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"
Jul 28 16:25:47.768730 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:25:47.768711    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495\": container with ID starting with 0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495 not found: ID does not exist" containerID="0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"
Jul 28 16:25:47.768775 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.768738    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495"} err="failed to get container status \"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495\": rpc error: code = NotFound desc = could not find container \"0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495\": container with ID starting with 0f06981fd3ea015eb95915fcc0339de2acaa4d49ba4ae74c0a0272ba377a2495 not found: ID does not exist"
Jul 28 16:25:47.768775 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.768755    2580 scope.go:117] "RemoveContainer" containerID="902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"
Jul 28 16:25:47.768984 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:25:47.768969    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a\": container with ID starting with 902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a not found: ID does not exist" containerID="902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"
Jul 28 16:25:47.769032 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.768987    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a"} err="failed to get container status \"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a\": rpc error: code = NotFound desc = could not find container \"902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a\": container with ID starting with 902928b2b4ad7ff13ff0df1d4e748837338a9d1d57b47b5bb34e84c8c5986c8a not found: ID does not exist"
Jul 28 16:25:47.769032 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.768999    2580 scope.go:117] "RemoveContainer" containerID="76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"
Jul 28 16:25:47.769238 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:25:47.769215    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0\": container with ID starting with 76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0 not found: ID does not exist" containerID="76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"
Jul 28 16:25:47.769285 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.769241    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0"} err="failed to get container status \"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0\": rpc error: code = NotFound desc = could not find container \"76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0\": container with ID starting with 76cbf7e0bc5f78239d5ec1e3d9449034729d6e693f0340c12d436a38eb76c9e0 not found: ID does not exist"
Jul 28 16:25:47.772732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:47.772713    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-lhgmk"]
Jul 28 16:25:48.185944 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.185909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" path="/var/lib/kubelet/pods/cce1ff07-bb57-47be-ba8a-dbf358472824/volumes"
Jul 28 16:25:48.609972 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.609938    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:25:48.659462 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659424    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:25:48.659801 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659787    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="storage-initializer"
Jul 28 16:25:48.659850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659803    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="storage-initializer"
Jul 28 16:25:48.659850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659816    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kube-rbac-proxy"
Jul 28 16:25:48.659850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kube-rbac-proxy"
Jul 28 16:25:48.659850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659841    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container"
Jul 28 16:25:48.659850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659848    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container"
Jul 28 16:25:48.660023 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659900    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kube-rbac-proxy"
Jul 28 16:25:48.660023 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.659908    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce1ff07-bb57-47be-ba8a-dbf358472824" containerName="kserve-container"
Jul 28 16:25:48.664612 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.664592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.667427 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667406    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 16:25:48.667551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:25:48.667551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 16:25:48.667551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:25:48.667774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667755    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:25:48.667817 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.667793    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:25:48.668722 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.668706    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:25:48.680384 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.680359    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:25:48.735358 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.735319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.735528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.735367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.735528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.735420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.735528 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.735441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.735637 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.735553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxzd\" (UniqueName: \"kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836216 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxzd\" (UniqueName: \"kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836410 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.836640 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:25:48.836447    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 28 16:25:48.836640 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:25:48.836528    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls podName:3937465e-9966-477a-b6fe-b0a1d4b1e5fc nodeName:}" failed. No retries permitted until 2026-07-28 16:25:49.336507011 +0000 UTC m=+3163.705159928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 28 16:25:48.836845 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.837009 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.836990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.837045 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.837016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.848476 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.848456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxzd\" (UniqueName: \"kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:48.974552 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:48.974477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:25:49.340545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:49.340515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:49.343078 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:49.343048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:49.575105 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:49.575064    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:49.709800 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:49.709777    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:25:49.712401 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:25:49.712375    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3937465e_9966_477a_b6fe_b0a1d4b1e5fc.slice/crio-f82430b75af7796b0ad2f7417d7ec2784ac07cc9e49b7319d1d6f8038b28a9ca WatchSource:0}: Error finding container f82430b75af7796b0ad2f7417d7ec2784ac07cc9e49b7319d1d6f8038b28a9ca: Status 404 returned error can't find the container with id f82430b75af7796b0ad2f7417d7ec2784ac07cc9e49b7319d1d6f8038b28a9ca
Jul 28 16:25:49.754179 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:49.754150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerStarted","Data":"f82430b75af7796b0ad2f7417d7ec2784ac07cc9e49b7319d1d6f8038b28a9ca"}
Jul 28 16:25:50.760372 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:50.760332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerStarted","Data":"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"}
Jul 28 16:25:53.775881 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:53.775846    2580 generic.go:356] "Generic (PLEG): container finished" podID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerID="ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251" exitCode=0
Jul 28 16:25:53.776285 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:53.775919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerDied","Data":"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"}
Jul 28 16:25:54.783260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.783218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerStarted","Data":"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"}
Jul 28 16:25:54.783664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.783287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerStarted","Data":"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"}
Jul 28 16:25:54.783664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.783559    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:54.783783 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.783712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:25:54.784938 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.784913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:25:54.808385 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:54.808330    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podStartSLOduration=3.051204155 podStartE2EDuration="6.808313638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757109483s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:25:54.806816807 +0000 UTC m=+3169.175469720" watchObservedRunningTime="2026-07-28 16:25:54.808313638 +0000 UTC m=+3169.176966553"
Jul 28 16:25:55.788645 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:25:55.788602    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:00.795549 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:00.795520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:26:00.796121 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:00.796091    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:10.796371 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:10.796333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:20.796215 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:20.796175    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:30.796952 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:30.796907    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:40.796994 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:40.796953    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:50.796696 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:50.796650    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:26:53.212310 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:26:53.212273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:27:00.796320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:00.796227    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:27:09.181488 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:09.181437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:27:19.182058 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:19.182020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:27:29.745783 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:29.745748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:27:29.746173 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:29.746000    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container" containerID="cri-o://1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0" gracePeriod=30
Jul 28 16:27:29.746173 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:29.746049    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kube-rbac-proxy" containerID="cri-o://813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc" gracePeriod=30
Jul 28 16:27:29.776300 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:29.776263    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:27:30.212324 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:30.212293    2580 generic.go:356] "Generic (PLEG): container finished" podID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerID="813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc" exitCode=2
Jul 28 16:27:30.212492 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:30.212349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerDied","Data":"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"}
Jul 28 16:27:30.789418 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:30.789369    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 16:27:33.492588 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.492563    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:27:33.634534 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert\") pod \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") "
Jul 28 16:27:33.634534 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") pod \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") "
Jul 28 16:27:33.634738 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config\") pod \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") "
Jul 28 16:27:33.634738 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634705    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwxzd\" (UniqueName: \"kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd\") pod \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") "
Jul 28 16:27:33.634819 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location\") pod \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\" (UID: \"3937465e-9966-477a-b6fe-b0a1d4b1e5fc\") "
Jul 28 16:27:33.634927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.634901    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3937465e-9966-477a-b6fe-b0a1d4b1e5fc" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:27:33.635146 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.635008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3937465e-9966-477a-b6fe-b0a1d4b1e5fc" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:27:33.635220 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.635108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3937465e-9966-477a-b6fe-b0a1d4b1e5fc" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:27:33.635220 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.635187    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:27:33.636800 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.636781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3937465e-9966-477a-b6fe-b0a1d4b1e5fc" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:27:33.636949 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.636927    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd" (OuterVolumeSpecName: "kube-api-access-vwxzd") pod "3937465e-9966-477a-b6fe-b0a1d4b1e5fc" (UID: "3937465e-9966-477a-b6fe-b0a1d4b1e5fc"). InnerVolumeSpecName "kube-api-access-vwxzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:27:33.736312 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.736280    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:27:33.736312 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.736310    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxzd\" (UniqueName: \"kubernetes.io/projected/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kube-api-access-vwxzd\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:27:33.736312 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.736319    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:27:33.736563 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:33.736328    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3937465e-9966-477a-b6fe-b0a1d4b1e5fc-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:27:34.231033 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.230999    2580 generic.go:356] "Generic (PLEG): container finished" podID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerID="1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0" exitCode=0
Jul 28 16:27:34.231228 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.231081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"
Jul 28 16:27:34.231228 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.231077    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerDied","Data":"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"}
Jul 28 16:27:34.231228 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.231202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m" event={"ID":"3937465e-9966-477a-b6fe-b0a1d4b1e5fc","Type":"ContainerDied","Data":"f82430b75af7796b0ad2f7417d7ec2784ac07cc9e49b7319d1d6f8038b28a9ca"}
Jul 28 16:27:34.231400 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.231234    2580 scope.go:117] "RemoveContainer" containerID="813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"
Jul 28 16:27:34.242054 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.242021    2580 scope.go:117] "RemoveContainer" containerID="1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"
Jul 28 16:27:34.250072 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.250054    2580 scope.go:117] "RemoveContainer" containerID="ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"
Jul 28 16:27:34.252819 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.252796    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:27:34.259724 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.259703    2580 scope.go:117] "RemoveContainer" containerID="813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"
Jul 28 16:27:34.259844 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.259823    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-zvb8m"]
Jul 28 16:27:34.260032 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:27:34.260015    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc\": container with ID starting with 813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc not found: ID does not exist" containerID="813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"
Jul 28 16:27:34.260083 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.260042    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc"} err="failed to get container status \"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc\": rpc error: code = NotFound desc = could not find container \"813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc\": container with ID starting with 813553971ab6fd6b437ea5e52c250c866d5521d9eefe2aee2e288160fa06cadc not found: ID does not exist"
Jul 28 16:27:34.260083 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.260061    2580 scope.go:117] "RemoveContainer" containerID="1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"
Jul 28 16:27:34.260404 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:27:34.260388    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0\": container with ID starting with 1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0 not found: ID does not exist" containerID="1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"
Jul 28 16:27:34.260452 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.260410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0"} err="failed to get container status \"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0\": rpc error: code = NotFound desc = could not find container \"1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0\": container with ID starting with 1a3bdd631b4f92d82a4265d11cb8dea74ccf5d24eba386f9d812d4064543e9d0 not found: ID does not exist"
Jul 28 16:27:34.260452 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.260428    2580 scope.go:117] "RemoveContainer" containerID="ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"
Jul 28 16:27:34.260656 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:27:34.260641    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251\": container with ID starting with ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251 not found: ID does not exist" containerID="ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"
Jul 28 16:27:34.260710 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.260658    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251"} err="failed to get container status \"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251\": rpc error: code = NotFound desc = could not find container \"ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251\": container with ID starting with ffe57dbe28a7b75897a321e49fc985539872253503bf7c48bc1ec2347579e251 not found: ID does not exist"
Jul 28 16:27:34.918338 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.918306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:27:34.988596 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.988555    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:27:34.989000 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.988981    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container"
Jul 28 16:27:34.989088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989002    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container"
Jul 28 16:27:34.989088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989020    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="storage-initializer"
Jul 28 16:27:34.989088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989028    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="storage-initializer"
Jul 28 16:27:34.989088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989043    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kube-rbac-proxy"
Jul 28 16:27:34.989088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989051    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kube-rbac-proxy"
Jul 28 16:27:34.989381 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989151    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kube-rbac-proxy"
Jul 28 16:27:34.989381 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.989168    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" containerName="kserve-container"
Jul 28 16:27:34.993775 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.993754    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:34.996689 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.996671    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:27:34.996796 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.996733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:27:34.997121 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.997102    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9f7712-kube-rbac-proxy-sar-config\""
Jul 28 16:27:34.997121 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.997137    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:27:34.997350 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.997106    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:27:34.998046 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.998030    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9f7712-predictor-serving-cert\""
Jul 28 16:27:34.998140 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:34.998092    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:27:35.005261 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.005239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:27:35.148886 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.148846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.149082 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.148915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.149082 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.149003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.149266 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.149103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.149266 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.149182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5c54\" (UniqueName: \"kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250484 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250484 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5c54\" (UniqueName: \"kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250484 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250795 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.250795 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:27:35.250535    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-9f7712-predictor-serving-cert: secret "isvc-primary-9f7712-predictor-serving-cert" not found
Jul 28 16:27:35.250795 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:27:35.250614    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls podName:54683907-9f62-43bc-8921-39ffdfd2eaf9 nodeName:}" failed. No retries permitted until 2026-07-28 16:27:35.750597783 +0000 UTC m=+3270.119250677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls") pod "isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9") : secret "isvc-primary-9f7712-predictor-serving-cert" not found
Jul 28 16:27:35.250986 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.250905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.251331 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.251312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.251379 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.251312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.263698 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.263670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5c54\" (UniqueName: \"kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.444693 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.444660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:27:35.755520 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.755483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.758685 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.758646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") pod \"isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") " pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:35.904836 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:35.904796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:36.077563 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:36.077536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:27:36.079226 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:27:36.079187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54683907_9f62_43bc_8921_39ffdfd2eaf9.slice/crio-9bed2a6cb9c6821693872ecf317aafa779b809ce2622645b2a27a7f3d7a7292d WatchSource:0}: Error finding container 9bed2a6cb9c6821693872ecf317aafa779b809ce2622645b2a27a7f3d7a7292d: Status 404 returned error can't find the container with id 9bed2a6cb9c6821693872ecf317aafa779b809ce2622645b2a27a7f3d7a7292d
Jul 28 16:27:36.186490 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:36.186453    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3937465e-9966-477a-b6fe-b0a1d4b1e5fc" path="/var/lib/kubelet/pods/3937465e-9966-477a-b6fe-b0a1d4b1e5fc/volumes"
Jul 28 16:27:36.242853 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:36.242815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerStarted","Data":"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"}
Jul 28 16:27:36.242991 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:36.242888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerStarted","Data":"9bed2a6cb9c6821693872ecf317aafa779b809ce2622645b2a27a7f3d7a7292d"}
Jul 28 16:27:40.261884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:40.261849    2580 generic.go:356] "Generic (PLEG): container finished" podID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerID="261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc" exitCode=0
Jul 28 16:27:40.262284 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:40.261917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerDied","Data":"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"}
Jul 28 16:27:41.269658 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:41.269620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerStarted","Data":"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"}
Jul 28 16:27:41.270079 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:41.269683    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerStarted","Data":"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"}
Jul 28 16:27:41.270079 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:41.269866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:41.294772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:41.294685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podStartSLOduration=3.7466971449999997 podStartE2EDuration="7.294667445s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.5479703s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:27:41.293012386 +0000 UTC m=+3275.661665302" watchObservedRunningTime="2026-07-28 16:27:41.294667445 +0000 UTC m=+3275.663320360"
Jul 28 16:27:42.276784 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:42.276751    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:42.278091 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:42.278064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:27:43.281409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:43.281360    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:27:48.285774 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:48.285745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:27:48.286391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:48.286364    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:27:58.286849 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:27:58.286803    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:08.286395 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:08.286350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:18.286728 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:18.286687    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:28.286357 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:28.286267    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:38.286409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:38.286365    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:48.286464 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:48.286428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:28:50.182089 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:28:50.182041    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:29:00.186222 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:00.186180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:29:20.186678 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.186649    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:20.190145 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.190113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.192907 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.192883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9f7712-predictor-serving-cert\""
Jul 28 16:29:20.192907 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.192888    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9f7712-kube-rbac-proxy-sar-config\""
Jul 28 16:29:20.193115 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.192956    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9f7712\""
Jul 28 16:29:20.193115 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.192958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9f7712-dockercfg-cjnkf\""
Jul 28 16:29:20.196571 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.196546    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:20.270496 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.270455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575hp\" (UniqueName: \"kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.270734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.270708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.270894 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.270795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.270962 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.270916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.271037 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.271012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575hp\" (UniqueName: \"kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.372975 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.372944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.373146 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.373105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.373216 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.373160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.375054 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.375022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.382348 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.382328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-575hp\" (UniqueName: \"kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp\") pod \"isvc-secondary-9f7712-predictor-6744d4d446-wclmp\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") " pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.501724 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.501630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:20.637891 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.637859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:20.640004 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:29:20.639974    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784cb35f_bbb9_4140_bab8_eb146011994a.slice/crio-99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f WatchSource:0}: Error finding container 99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f: Status 404 returned error can't find the container with id 99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f
Jul 28 16:29:20.718160 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.718109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerStarted","Data":"094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"}
Jul 28 16:29:20.718339 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.718198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerStarted","Data":"99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f"}
Jul 28 16:29:20.893260 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.893221    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:20.978525 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:20.978487    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:24.738038 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:24.737965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/0.log"
Jul 28 16:29:24.738038 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:24.738014    2580 generic.go:356] "Generic (PLEG): container finished" podID="784cb35f-bbb9-4140-bab8-eb146011994a" containerID="094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd" exitCode=1
Jul 28 16:29:24.738551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:24.738107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerDied","Data":"094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"}
Jul 28 16:29:25.744743 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:25.744713    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/0.log"
Jul 28 16:29:25.745145 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:25.744768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerStarted","Data":"9791379f74791ebf51a17b8514018a91b8df1e31d5574ef0eb07171515a69583"}
Jul 28 16:29:27.755527 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.755492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/1.log"
Jul 28 16:29:27.755954 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.755871    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/0.log"
Jul 28 16:29:27.755954 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.755908    2580 generic.go:356] "Generic (PLEG): container finished" podID="784cb35f-bbb9-4140-bab8-eb146011994a" containerID="9791379f74791ebf51a17b8514018a91b8df1e31d5574ef0eb07171515a69583" exitCode=1
Jul 28 16:29:27.755954 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.755942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerDied","Data":"9791379f74791ebf51a17b8514018a91b8df1e31d5574ef0eb07171515a69583"}
Jul 28 16:29:27.756942 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.755996    2580 scope.go:117] "RemoveContainer" containerID="094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"
Jul 28 16:29:27.757020 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.757011    2580 scope.go:117] "RemoveContainer" containerID="094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"
Jul 28 16:29:27.768015 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:29:27.767990    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9f7712-predictor-6744d4d446-wclmp_kserve-ci-e2e-test(784cb35f-bbb9-4140-bab8-eb146011994a)\"" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" podUID="784cb35f-bbb9-4140-bab8-eb146011994a"
Jul 28 16:29:27.768267 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:29:27.768239    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_kserve-ci-e2e-test_784cb35f-bbb9-4140-bab8-eb146011994a_0 in pod sandbox 99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f from index: no such id: '094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd'" containerID="094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"
Jul 28 16:29:27.768337 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:27.768279    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_kserve-ci-e2e-test_784cb35f-bbb9-4140-bab8-eb146011994a_0 in pod sandbox 99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f from index: no such id: '094cb09ebd11a1eeac2f61d23f4f9644288ab99fcce7b4d966a0220dd06475fd'"
Jul 28 16:29:28.768653 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:28.768626    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/1.log"
Jul 28 16:29:36.233916 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.233883    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:36.257325 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.257297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:36.308064 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.308029    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:29:36.308349 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.308322    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" containerID="cri-o://a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba" gracePeriod=30
Jul 28 16:29:36.308428 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.308344    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kube-rbac-proxy" containerID="cri-o://de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9" gracePeriod=30
Jul 28 16:29:36.332614 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.332582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:36.429722 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.429702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/1.log"
Jul 28 16:29:36.431763 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.431746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:36.518294 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config\") pod \"784cb35f-bbb9-4140-bab8-eb146011994a\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") "
Jul 28 16:29:36.518294 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert\") pod \"784cb35f-bbb9-4140-bab8-eb146011994a\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") "
Jul 28 16:29:36.518294 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-575hp\" (UniqueName: \"kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp\") pod \"784cb35f-bbb9-4140-bab8-eb146011994a\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") "
Jul 28 16:29:36.518579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls\") pod \"784cb35f-bbb9-4140-bab8-eb146011994a\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") "
Jul 28 16:29:36.518579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518351    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location\") pod \"784cb35f-bbb9-4140-bab8-eb146011994a\" (UID: \"784cb35f-bbb9-4140-bab8-eb146011994a\") "
Jul 28 16:29:36.518695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518666    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "784cb35f-bbb9-4140-bab8-eb146011994a" (UID: "784cb35f-bbb9-4140-bab8-eb146011994a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:29:36.518695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "784cb35f-bbb9-4140-bab8-eb146011994a" (UID: "784cb35f-bbb9-4140-bab8-eb146011994a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:29:36.518772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.518668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "784cb35f-bbb9-4140-bab8-eb146011994a" (UID: "784cb35f-bbb9-4140-bab8-eb146011994a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:29:36.520577 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.520555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp" (OuterVolumeSpecName: "kube-api-access-575hp") pod "784cb35f-bbb9-4140-bab8-eb146011994a" (UID: "784cb35f-bbb9-4140-bab8-eb146011994a"). InnerVolumeSpecName "kube-api-access-575hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:29:36.520668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.520605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "784cb35f-bbb9-4140-bab8-eb146011994a" (UID: "784cb35f-bbb9-4140-bab8-eb146011994a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:29:36.619441 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.619393    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:36.619441 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.619434    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-575hp\" (UniqueName: \"kubernetes.io/projected/784cb35f-bbb9-4140-bab8-eb146011994a-kube-api-access-575hp\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:36.619441 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.619445    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784cb35f-bbb9-4140-bab8-eb146011994a-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:36.619441 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.619457    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784cb35f-bbb9-4140-bab8-eb146011994a-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:36.619734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.619465    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784cb35f-bbb9-4140-bab8-eb146011994a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:36.809314 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.809230    2580 generic.go:356] "Generic (PLEG): container finished" podID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerID="de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9" exitCode=2
Jul 28 16:29:36.809472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.809307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerDied","Data":"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"}
Jul 28 16:29:36.810550 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.810535    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9f7712-predictor-6744d4d446-wclmp_784cb35f-bbb9-4140-bab8-eb146011994a/storage-initializer/1.log"
Jul 28 16:29:36.810632 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.810617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp" event={"ID":"784cb35f-bbb9-4140-bab8-eb146011994a","Type":"ContainerDied","Data":"99921510299ebb0e77a0470ea556508da3e752802e09d8051fdfe0966592b53f"}
Jul 28 16:29:36.810673 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.810633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"
Jul 28 16:29:36.810709 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.810681    2580 scope.go:117] "RemoveContainer" containerID="9791379f74791ebf51a17b8514018a91b8df1e31d5574ef0eb07171515a69583"
Jul 28 16:29:36.849399 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.849363    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:36.854651 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:36.854621    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9f7712-predictor-6744d4d446-wclmp"]
Jul 28 16:29:38.186602 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:38.186557    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" path="/var/lib/kubelet/pods/784cb35f-bbb9-4140-bab8-eb146011994a/volumes"
Jul 28 16:29:38.282074 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:38.282031    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 28 16:29:40.182641 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:40.182599    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:29:41.163031 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.163005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:29:41.258712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.258626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert\") pod \"54683907-9f62-43bc-8921-39ffdfd2eaf9\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") "
Jul 28 16:29:41.258712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.258664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") pod \"54683907-9f62-43bc-8921-39ffdfd2eaf9\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") "
Jul 28 16:29:41.258712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.258687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location\") pod \"54683907-9f62-43bc-8921-39ffdfd2eaf9\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") "
Jul 28 16:29:41.258712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.258715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config\") pod \"54683907-9f62-43bc-8921-39ffdfd2eaf9\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") "
Jul 28 16:29:41.259374 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.258753    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5c54\" (UniqueName: \"kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54\") pod \"54683907-9f62-43bc-8921-39ffdfd2eaf9\" (UID: \"54683907-9f62-43bc-8921-39ffdfd2eaf9\") "
Jul 28 16:29:41.259374 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.259147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54683907-9f62-43bc-8921-39ffdfd2eaf9" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:29:41.259374 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.259089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54683907-9f62-43bc-8921-39ffdfd2eaf9" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:29:41.259374 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.259242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54683907-9f62-43bc-8921-39ffdfd2eaf9" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:29:41.260947 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.260923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54683907-9f62-43bc-8921-39ffdfd2eaf9" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:29:41.261192 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.261170    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54" (OuterVolumeSpecName: "kube-api-access-c5c54") pod "54683907-9f62-43bc-8921-39ffdfd2eaf9" (UID: "54683907-9f62-43bc-8921-39ffdfd2eaf9"). InnerVolumeSpecName "kube-api-access-c5c54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:29:41.359371 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.359336    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:41.359371 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.359364    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54683907-9f62-43bc-8921-39ffdfd2eaf9-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:41.359371 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.359374    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54683907-9f62-43bc-8921-39ffdfd2eaf9-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:41.359599 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.359384    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:41.359599 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.359393    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5c54\" (UniqueName: \"kubernetes.io/projected/54683907-9f62-43bc-8921-39ffdfd2eaf9-kube-api-access-c5c54\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:29:41.436013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.435982    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:41.696559 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.696526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:41.834940 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.834907    2580 generic.go:356] "Generic (PLEG): container finished" podID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerID="a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba" exitCode=0
Jul 28 16:29:41.835168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.834979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"
Jul 28 16:29:41.835168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.834998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerDied","Data":"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"}
Jul 28 16:29:41.835168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.835068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf" event={"ID":"54683907-9f62-43bc-8921-39ffdfd2eaf9","Type":"ContainerDied","Data":"9bed2a6cb9c6821693872ecf317aafa779b809ce2622645b2a27a7f3d7a7292d"}
Jul 28 16:29:41.835168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.835096    2580 scope.go:117] "RemoveContainer" containerID="de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"
Jul 28 16:29:41.844718 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.844698    2580 scope.go:117] "RemoveContainer" containerID="a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"
Jul 28 16:29:41.852589 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.852572    2580 scope.go:117] "RemoveContainer" containerID="261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"
Jul 28 16:29:41.858747 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.858718    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:29:41.860624 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.860610    2580 scope.go:117] "RemoveContainer" containerID="de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"
Jul 28 16:29:41.860890 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:29:41.860872    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9\": container with ID starting with de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9 not found: ID does not exist" containerID="de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"
Jul 28 16:29:41.860929 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.860899    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9"} err="failed to get container status \"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9\": rpc error: code = NotFound desc = could not find container \"de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9\": container with ID starting with de250b46675ebc94e58ccca8c1e62bad0a5a5daf32e88f906d22c8eae8a985c9 not found: ID does not exist"
Jul 28 16:29:41.860929 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.860919    2580 scope.go:117] "RemoveContainer" containerID="a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"
Jul 28 16:29:41.861173 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:29:41.861157    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba\": container with ID starting with a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba not found: ID does not exist" containerID="a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"
Jul 28 16:29:41.861227 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.861180    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba"} err="failed to get container status \"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba\": rpc error: code = NotFound desc = could not find container \"a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba\": container with ID starting with a1e4dc926499bdabcb2ed69cde70cda694c2ba504de1ecac1efad772b4032aba not found: ID does not exist"
Jul 28 16:29:41.861227 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.861195    2580 scope.go:117] "RemoveContainer" containerID="261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"
Jul 28 16:29:41.861430 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:29:41.861411    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc\": container with ID starting with 261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc not found: ID does not exist" containerID="261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"
Jul 28 16:29:41.861489 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.861438    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc"} err="failed to get container status \"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc\": rpc error: code = NotFound desc = could not find container \"261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc\": container with ID starting with 261bcf2a3164adfc7cfa8e9b4aef4751ba29404e0235a505c99cc734a32b90dc not found: ID does not exist"
Jul 28 16:29:41.865643 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:41.865622    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9f7712-predictor-7ffdfcdc49-cf9cf"]
Jul 28 16:29:42.186545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:42.186514    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" path="/var/lib/kubelet/pods/54683907-9f62-43bc-8921-39ffdfd2eaf9/volumes"
Jul 28 16:29:46.525287 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:46.525253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:51.687604 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:51.687565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:29:52.039546 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:29:52.039445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:31:16.354378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:31:16.354342    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:31:41.919725 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:31:41.919645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:31:46.999389 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:31:46.999355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:31:47.307423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:31:47.307333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:32:31.330758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:32:31.330723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:32:32.710392 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:32:32.710355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:33:11.376366 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:11.376324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:33:17.248804 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:17.248757    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:33:22.349876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.349831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:33:22.450356 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450316    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:33:22.450730 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450712    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450732    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450750    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450758    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450780    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450790    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450802    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450810    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450821    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kube-rbac-proxy"
Jul 28 16:33:22.450830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kube-rbac-proxy"
Jul 28 16:33:22.451320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450915    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kserve-container"
Jul 28 16:33:22.451320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450933    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="54683907-9f62-43bc-8921-39ffdfd2eaf9" containerName="kube-rbac-proxy"
Jul 28 16:33:22.451320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450942    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.451320 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.450951    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="784cb35f-bbb9-4140-bab8-eb146011994a" containerName="storage-initializer"
Jul 28 16:33:22.454318 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.454297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.457153 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.457110    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:33:22.457355 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.457315    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:33:22.457355 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.457344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:33:22.457507 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.457343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:33:22.457507 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.457438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:33:22.458261 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.458244    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 16:33:22.458346 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.458296    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 16:33:22.465067 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.465041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:33:22.582741 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.582710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.582927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.582758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftj25\" (UniqueName: \"kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.582927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.582803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.582927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.582824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.582927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.582911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.594005 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.593974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:33:22.683909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.683828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.683909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.683878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684226 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.683912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftj25\" (UniqueName: \"kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684226 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.683943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684226 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.684028    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684401 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.684382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684656 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.684635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.684697 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.684641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.686469 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.686449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.693025 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.693000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftj25\" (UniqueName: \"kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.765758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.765721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:22.897933 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:22.897905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:33:22.900108 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:33:22.900066    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f79b5e_39a7_4f82_87a7_ad490b738dd8.slice/crio-68af969094d00984949df3d80375236196c1d45ac74ed4c20efdc387df46bfa5 WatchSource:0}: Error finding container 68af969094d00984949df3d80375236196c1d45ac74ed4c20efdc387df46bfa5: Status 404 returned error can't find the container with id 68af969094d00984949df3d80375236196c1d45ac74ed4c20efdc387df46bfa5
Jul 28 16:33:23.767301 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:23.767260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerStarted","Data":"d8d6b52b41fb7099057467ee8ff895919accf8e4e2808964f49acf7625e09e3d"}
Jul 28 16:33:23.767828 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:23.767370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerStarted","Data":"68af969094d00984949df3d80375236196c1d45ac74ed4c20efdc387df46bfa5"}
Jul 28 16:33:27.787614 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:27.787581    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerID="d8d6b52b41fb7099057467ee8ff895919accf8e4e2808964f49acf7625e09e3d" exitCode=0
Jul 28 16:33:27.787999 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:27.787632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerDied","Data":"d8d6b52b41fb7099057467ee8ff895919accf8e4e2808964f49acf7625e09e3d"}
Jul 28 16:33:27.788970 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:27.788951    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:33:50.911867 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:50.911824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerStarted","Data":"81c16eac2533de5e23dfe3a291c57cc295f360fbd0a36eb80ede278620742d00"}
Jul 28 16:33:50.912402 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:50.911924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerStarted","Data":"344400f304516b6ba0b53172d7e6666571627d441d72542112f70f1a18fef814"}
Jul 28 16:33:50.912402 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:50.912159    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:50.935001 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:50.934949    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podStartSLOduration=2.585429585 podStartE2EDuration="28.934936248s" totalImagesPullingTime="22.420492812s" totalInitContainerRuntime="3.929013851s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:33:50.933441472 +0000 UTC m=+3645.302094388" watchObservedRunningTime="2026-07-28 16:33:50.934936248 +0000 UTC m=+3645.303589164"
Jul 28 16:33:51.916182 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:51.916149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:51.917249 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:51.917221    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:52.921300 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:52.921261    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:57.926092 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:57.926058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:33:57.926694 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:33:57.926669    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:07.926762 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:07.926714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:17.927627 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:17.927582    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:27.927172 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:27.927077    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:37.927396 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:37.927354    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:47.926644 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:47.926601    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:34:57.926843 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:34:57.926800    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:35:07.928122 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:07.928084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:35:12.489202 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:12.489170    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:35:12.489731 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:12.489395    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" containerID="cri-o://344400f304516b6ba0b53172d7e6666571627d441d72542112f70f1a18fef814" gracePeriod=30
Jul 28 16:35:12.489731 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:12.489432    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy" containerID="cri-o://81c16eac2533de5e23dfe3a291c57cc295f360fbd0a36eb80ede278620742d00" gracePeriod=30
Jul 28 16:35:12.543638 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:12.543604    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:35:12.921876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:12.921840    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 16:35:13.270850 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:13.270776    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerID="81c16eac2533de5e23dfe3a291c57cc295f360fbd0a36eb80ede278620742d00" exitCode=2
Jul 28 16:35:13.270998 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:13.270856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerDied","Data":"81c16eac2533de5e23dfe3a291c57cc295f360fbd0a36eb80ede278620742d00"}
Jul 28 16:35:17.645188 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.645149    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:35:17.710659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.710624    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:35:17.714378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.714361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.717404 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.717382    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 16:35:17.717537 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.717390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 16:35:17.724597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.724572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:35:17.808479 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.808443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.808690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.808507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.808690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.808548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.808690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.808578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.808690 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.808630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznr6\" (UniqueName: \"kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznr6\" (UniqueName: \"kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.909895 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:35:17.909768    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 16:35:17.909895 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:35:17.909848    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls podName:adbb48ea-cdef-438c-ad4b-4420665d998c nodeName:}" failed. No retries permitted until 2026-07-28 16:35:18.409826026 +0000 UTC m=+3732.778478920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 16:35:17.910105 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.909945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.910323 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.910306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.910363 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.910345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:17.921905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.921867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 16:35:17.926859 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.926832    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:35:17.934249 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:17.934225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznr6\" (UniqueName: \"kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:18.023887 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.023853    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:35:18.297824 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.297746    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerID="344400f304516b6ba0b53172d7e6666571627d441d72542112f70f1a18fef814" exitCode=0
Jul 28 16:35:18.297973 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.297822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerDied","Data":"344400f304516b6ba0b53172d7e6666571627d441d72542112f70f1a18fef814"}
Jul 28 16:35:18.415634 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.415610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:18.418007 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.417988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:18.436835 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.436810    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:35:18.516807 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.516766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls\") pod \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") "
Jul 28 16:35:18.516807 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.516809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location\") pod \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") "
Jul 28 16:35:18.517026 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.516833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftj25\" (UniqueName: \"kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25\") pod \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") "
Jul 28 16:35:18.517026 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.516851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config\") pod \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") "
Jul 28 16:35:18.517026 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.516915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert\") pod \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\" (UID: \"a0f79b5e-39a7-4f82-87a7-ad490b738dd8\") "
Jul 28 16:35:18.517287 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.517261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0f79b5e-39a7-4f82-87a7-ad490b738dd8" (UID: "a0f79b5e-39a7-4f82-87a7-ad490b738dd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:35:18.517358 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.517261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0f79b5e-39a7-4f82-87a7-ad490b738dd8" (UID: "a0f79b5e-39a7-4f82-87a7-ad490b738dd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:18.517358 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.517304    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0f79b5e-39a7-4f82-87a7-ad490b738dd8" (UID: "a0f79b5e-39a7-4f82-87a7-ad490b738dd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:18.519158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.519115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0f79b5e-39a7-4f82-87a7-ad490b738dd8" (UID: "a0f79b5e-39a7-4f82-87a7-ad490b738dd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:35:18.519158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.519153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25" (OuterVolumeSpecName: "kube-api-access-ftj25") pod "a0f79b5e-39a7-4f82-87a7-ad490b738dd8" (UID: "a0f79b5e-39a7-4f82-87a7-ad490b738dd8"). InnerVolumeSpecName "kube-api-access-ftj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:35:18.617934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.617902    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:35:18.617934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.617930    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:35:18.617934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.617939    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftj25\" (UniqueName: \"kubernetes.io/projected/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-api-access-ftj25\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:35:18.618211 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.617948    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:35:18.618211 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.617957    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f79b5e-39a7-4f82-87a7-ad490b738dd8-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:35:18.624873 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.624850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:18.764013 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:18.763989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:35:18.765854 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:35:18.765828    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbb48ea_cdef_438c_ad4b_4420665d998c.slice/crio-def73ed2a6570576288b046fa33dd5946ccbcdf547f63250d0eec473b93143b4 WatchSource:0}: Error finding container def73ed2a6570576288b046fa33dd5946ccbcdf547f63250d0eec473b93143b4: Status 404 returned error can't find the container with id def73ed2a6570576288b046fa33dd5946ccbcdf547f63250d0eec473b93143b4
Jul 28 16:35:19.305930 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.305898    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"
Jul 28 16:35:19.305930 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.305909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn" event={"ID":"a0f79b5e-39a7-4f82-87a7-ad490b738dd8","Type":"ContainerDied","Data":"68af969094d00984949df3d80375236196c1d45ac74ed4c20efdc387df46bfa5"}
Jul 28 16:35:19.306217 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.305993    2580 scope.go:117] "RemoveContainer" containerID="81c16eac2533de5e23dfe3a291c57cc295f360fbd0a36eb80ede278620742d00"
Jul 28 16:35:19.307893 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.307867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerStarted","Data":"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"}
Jul 28 16:35:19.308009 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.307939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerStarted","Data":"def73ed2a6570576288b046fa33dd5946ccbcdf547f63250d0eec473b93143b4"}
Jul 28 16:35:19.314932 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.314914    2580 scope.go:117] "RemoveContainer" containerID="344400f304516b6ba0b53172d7e6666571627d441d72542112f70f1a18fef814"
Jul 28 16:35:19.324724 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.324706    2580 scope.go:117] "RemoveContainer" containerID="d8d6b52b41fb7099057467ee8ff895919accf8e4e2808964f49acf7625e09e3d"
Jul 28 16:35:19.341471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.341447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:35:19.346514 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:19.346492    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-x9rfn"]
Jul 28 16:35:20.186494 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:20.186464    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" path="/var/lib/kubelet/pods/a0f79b5e-39a7-4f82-87a7-ad490b738dd8/volumes"
Jul 28 16:35:23.328240 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:23.328209    2580 generic.go:356] "Generic (PLEG): container finished" podID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerID="6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7" exitCode=0
Jul 28 16:35:23.328702 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:23.328290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerDied","Data":"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"}
Jul 28 16:35:24.336076 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:24.336039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerStarted","Data":"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"}
Jul 28 16:35:24.336497 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:24.336116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerStarted","Data":"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"}
Jul 28 16:35:24.336497 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:24.336468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:24.336587 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:24.336517    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:35:24.359344 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:24.359295    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podStartSLOduration=3.737408269 podStartE2EDuration="7.359279283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.621871014s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:35:24.357456469 +0000 UTC m=+3738.726109389" watchObservedRunningTime="2026-07-28 16:35:24.359279283 +0000 UTC m=+3738.727932199"
Jul 28 16:35:30.346314 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:35:30.346285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:36:00.347331 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:00.347251    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 16:36:10.346895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:10.346855    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 16:36:20.347273 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:20.347231    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 16:36:30.347237 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:30.347197    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 16:36:40.369951 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:40.369916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:36:48.710230 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:48.710196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:36:48.710695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:48.710440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" containerID="cri-o://95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39" gracePeriod=30
Jul 28 16:36:48.710695 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:48.710502    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kube-rbac-proxy" containerID="cri-o://dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8" gracePeriod=30
Jul 28 16:36:48.792489 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:48.792454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:36:49.718466 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:49.718436    2580 generic.go:356] "Generic (PLEG): container finished" podID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerID="dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8" exitCode=2
Jul 28 16:36:49.718846 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:49.718508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerDied","Data":"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"}
Jul 28 16:36:50.341834 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:50.341781    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 28 16:36:50.347380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:50.347349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 28 16:36:53.660092 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.660068    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:36:53.741431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.741334    2580 generic.go:356] "Generic (PLEG): container finished" podID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerID="95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39" exitCode=0
Jul 28 16:36:53.741431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.741411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"
Jul 28 16:36:53.741431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.741416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerDied","Data":"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"}
Jul 28 16:36:53.741677 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.741500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj" event={"ID":"adbb48ea-cdef-438c-ad4b-4420665d998c","Type":"ContainerDied","Data":"def73ed2a6570576288b046fa33dd5946ccbcdf547f63250d0eec473b93143b4"}
Jul 28 16:36:53.741677 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.741532    2580 scope.go:117] "RemoveContainer" containerID="dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"
Jul 28 16:36:53.750691 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.750670    2580 scope.go:117] "RemoveContainer" containerID="95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"
Jul 28 16:36:53.756145 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert\") pod \"adbb48ea-cdef-438c-ad4b-4420665d998c\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") "
Jul 28 16:36:53.756259 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location\") pod \"adbb48ea-cdef-438c-ad4b-4420665d998c\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") "
Jul 28 16:36:53.756309 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznr6\" (UniqueName: \"kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6\") pod \"adbb48ea-cdef-438c-ad4b-4420665d998c\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") "
Jul 28 16:36:53.756352 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756334    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") pod \"adbb48ea-cdef-438c-ad4b-4420665d998c\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") "
Jul 28 16:36:53.756398 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config\") pod \"adbb48ea-cdef-438c-ad4b-4420665d998c\" (UID: \"adbb48ea-cdef-438c-ad4b-4420665d998c\") "
Jul 28 16:36:53.756766 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "adbb48ea-cdef-438c-ad4b-4420665d998c" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:36:53.756766 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "adbb48ea-cdef-438c-ad4b-4420665d998c" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:36:53.756922 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.756882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "adbb48ea-cdef-438c-ad4b-4420665d998c" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:36:53.757167 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.757109    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adbb48ea-cdef-438c-ad4b-4420665d998c-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:36:53.757247 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.757172    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:36:53.757247 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.757185    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adbb48ea-cdef-438c-ad4b-4420665d998c-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:36:53.758643 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.758612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adbb48ea-cdef-438c-ad4b-4420665d998c" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:36:53.759058 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.759033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6" (OuterVolumeSpecName: "kube-api-access-wznr6") pod "adbb48ea-cdef-438c-ad4b-4420665d998c" (UID: "adbb48ea-cdef-438c-ad4b-4420665d998c"). InnerVolumeSpecName "kube-api-access-wznr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:36:53.759344 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.759326    2580 scope.go:117] "RemoveContainer" containerID="6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"
Jul 28 16:36:53.773920 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.773903    2580 scope.go:117] "RemoveContainer" containerID="dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"
Jul 28 16:36:53.774229 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:36:53.774204    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8\": container with ID starting with dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8 not found: ID does not exist" containerID="dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"
Jul 28 16:36:53.774296 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.774240    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8"} err="failed to get container status \"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8\": rpc error: code = NotFound desc = could not find container \"dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8\": container with ID starting with dc6c0989f16dc9425516366832264abbf67cc65924043dbe40ebeb32cdafa1d8 not found: ID does not exist"
Jul 28 16:36:53.774296 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.774259    2580 scope.go:117] "RemoveContainer" containerID="95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"
Jul 28 16:36:53.774515 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:36:53.774498    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39\": container with ID starting with 95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39 not found: ID does not exist" containerID="95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"
Jul 28 16:36:53.774585 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.774524    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39"} err="failed to get container status \"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39\": rpc error: code = NotFound desc = could not find container \"95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39\": container with ID starting with 95ba215ec03b64c0bc1800e45bcdf9040912f40403208ef8465db512e53dbe39 not found: ID does not exist"
Jul 28 16:36:53.774585 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.774546    2580 scope.go:117] "RemoveContainer" containerID="6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"
Jul 28 16:36:53.774776 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:36:53.774758    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7\": container with ID starting with 6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7 not found: ID does not exist" containerID="6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"
Jul 28 16:36:53.774863 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.774778    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7"} err="failed to get container status \"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7\": rpc error: code = NotFound desc = could not find container \"6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7\": container with ID starting with 6b3e7422922826848156280d361894b4df786092867e2584d280b9e505ed3db7 not found: ID does not exist"
Jul 28 16:36:53.794790 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.794763    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:36:53.842235 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842201    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:36:53.842551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842530    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy"
Jul 28 16:36:53.842551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842546    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy"
Jul 28 16:36:53.842551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="storage-initializer"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842564    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="storage-initializer"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842575    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842582    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842594    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="storage-initializer"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="storage-initializer"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kube-rbac-proxy"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842614    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kube-rbac-proxy"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842628    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842681    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kube-rbac-proxy"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842690    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f79b5e-39a7-4f82-87a7-ad490b738dd8" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842696    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kserve-container"
Jul 28 16:36:53.842748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.842703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" containerName="kube-rbac-proxy"
Jul 28 16:36:53.847401 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.847384    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.849857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.849834    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 16:36:53.849972 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.849956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 16:36:53.857181 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857156    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:36:53.857591 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.857681 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.857744 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.857790 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.857824 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857806    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmml\" (UniqueName: \"kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.857906 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857892    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznr6\" (UniqueName: \"kubernetes.io/projected/adbb48ea-cdef-438c-ad4b-4420665d998c-kube-api-access-wznr6\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:36:53.857945 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.857913    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adbb48ea-cdef-438c-ad4b-4420665d998c-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:36:53.958562 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.958525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.958732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.958577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.958732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.958612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.958732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.958629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.958732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.958662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmml\" (UniqueName: \"kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.958975 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:36:53.958778    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 16:36:53.958975 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:36:53.958856    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls podName:2a2c7d38-f524-43ea-a4a8-f7def96db5c9 nodeName:}" failed. No retries permitted until 2026-07-28 16:36:54.458833206 +0000 UTC m=+3828.827486118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 16:36:53.959090 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.959041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.959361 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.959341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.959482 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.959462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:53.967942 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:53.967919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmml\" (UniqueName: \"kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:54.064976 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.064947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:36:54.069758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.069732    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-8ngxj"]
Jul 28 16:36:54.186109 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.186074    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbb48ea-cdef-438c-ad4b-4420665d998c" path="/var/lib/kubelet/pods/adbb48ea-cdef-438c-ad4b-4420665d998c/volumes"
Jul 28 16:36:54.317731 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.317637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:36:54.462094 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.462061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:54.464562 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.464537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:54.758444 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.758406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:36:54.895792 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:54.895759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:36:54.898610 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:36:54.898577    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2c7d38_f524_43ea_a4a8_f7def96db5c9.slice/crio-6a310b46ed6cb32ca5596281e6f8970e86c3f55ac25c34c01b95d70217417310 WatchSource:0}: Error finding container 6a310b46ed6cb32ca5596281e6f8970e86c3f55ac25c34c01b95d70217417310: Status 404 returned error can't find the container with id 6a310b46ed6cb32ca5596281e6f8970e86c3f55ac25c34c01b95d70217417310
Jul 28 16:36:55.758088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:55.758054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerStarted","Data":"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"}
Jul 28 16:36:55.758300 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:55.758113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerStarted","Data":"6a310b46ed6cb32ca5596281e6f8970e86c3f55ac25c34c01b95d70217417310"}
Jul 28 16:36:59.777709 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:59.777673    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerID="159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898" exitCode=0
Jul 28 16:36:59.778107 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:36:59.777717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerDied","Data":"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"}
Jul 28 16:37:00.791395 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:00.791357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerStarted","Data":"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"}
Jul 28 16:37:00.791782 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:00.791438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerStarted","Data":"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"}
Jul 28 16:37:00.791782 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:00.791623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:37:00.814846 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:00.814793    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podStartSLOduration=3.968247238 podStartE2EDuration="7.814777599s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.846530361s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:37:00.812732743 +0000 UTC m=+3835.181385660" watchObservedRunningTime="2026-07-28 16:37:00.814777599 +0000 UTC m=+3835.183430515"
Jul 28 16:37:01.796020 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:01.795991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:37:07.806200 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:07.806161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:37:37.807816 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:37.807714    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 16:37:47.806928 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:47.806889    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 16:37:57.807703 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:37:57.807658    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 16:38:07.807644 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:07.807592    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 16:38:17.868804 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:17.868760    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:38:23.978556 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:23.978517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:38:23.979028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:23.978793    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" containerID="cri-o://48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339" gracePeriod=30
Jul 28 16:38:23.979028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:23.978821    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kube-rbac-proxy" containerID="cri-o://9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672" gracePeriod=30
Jul 28 16:38:24.015595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:24.015561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:38:24.157392 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:24.157360    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerID="9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672" exitCode=2
Jul 28 16:38:24.157549 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:24.157410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerDied","Data":"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"}
Jul 28 16:38:27.800628 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:27.800581    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 28 16:38:27.807045 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:27.807012    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 16:38:28.718720 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.718692    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:38:28.899250 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert\") pod \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") "
Jul 28 16:38:28.899250 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899259    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmml\" (UniqueName: \"kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml\") pod \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") "
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location\") pod \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") "
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config\") pod \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") "
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") pod \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\" (UID: \"2a2c7d38-f524-43ea-a4a8-f7def96db5c9\") "
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a2c7d38-f524-43ea-a4a8-f7def96db5c9" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a2c7d38-f524-43ea-a4a8-f7def96db5c9" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a2c7d38-f524-43ea-a4a8-f7def96db5c9" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899844    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899864    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:38:28.899905 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.899879    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:38:28.901707 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.901684    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a2c7d38-f524-43ea-a4a8-f7def96db5c9" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:38:28.901707 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:28.901700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml" (OuterVolumeSpecName: "kube-api-access-jhmml") pod "2a2c7d38-f524-43ea-a4a8-f7def96db5c9" (UID: "2a2c7d38-f524-43ea-a4a8-f7def96db5c9"). InnerVolumeSpecName "kube-api-access-jhmml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:38:29.000780 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.000737    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmml\" (UniqueName: \"kubernetes.io/projected/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-kube-api-access-jhmml\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:38:29.000780 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.000767    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2c7d38-f524-43ea-a4a8-f7def96db5c9-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:38:29.094440 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.094398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:38:29.181991 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.181899    2580 generic.go:356] "Generic (PLEG): container finished" podID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerID="48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339" exitCode=0
Jul 28 16:38:29.181991 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.181952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"
Jul 28 16:38:29.182199 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.181981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerDied","Data":"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"}
Jul 28 16:38:29.182199 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.182040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp" event={"ID":"2a2c7d38-f524-43ea-a4a8-f7def96db5c9","Type":"ContainerDied","Data":"6a310b46ed6cb32ca5596281e6f8970e86c3f55ac25c34c01b95d70217417310"}
Jul 28 16:38:29.182199 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.182071    2580 scope.go:117] "RemoveContainer" containerID="9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"
Jul 28 16:38:29.191550 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.191528    2580 scope.go:117] "RemoveContainer" containerID="48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"
Jul 28 16:38:29.200143 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.200110    2580 scope.go:117] "RemoveContainer" containerID="159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"
Jul 28 16:38:29.208890 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.208871    2580 scope.go:117] "RemoveContainer" containerID="9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"
Jul 28 16:38:29.209281 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:38:29.209257    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672\": container with ID starting with 9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672 not found: ID does not exist" containerID="9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"
Jul 28 16:38:29.209341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209293    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672"} err="failed to get container status \"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672\": rpc error: code = NotFound desc = could not find container \"9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672\": container with ID starting with 9ea6e997d7a7d4b20239ea8e8316861f8626f44af3ad2e811e54e58f7c3e0672 not found: ID does not exist"
Jul 28 16:38:29.209341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209311    2580 scope.go:117] "RemoveContainer" containerID="48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"
Jul 28 16:38:29.209595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209577    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:38:29.209634 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:38:29.209599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339\": container with ID starting with 48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339 not found: ID does not exist" containerID="48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"
Jul 28 16:38:29.209634 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209613    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339"} err="failed to get container status \"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339\": rpc error: code = NotFound desc = could not find container \"48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339\": container with ID starting with 48c66aced0dcea2f48d317026af017bd8c89bb47f322513ae29a1eee1f617339 not found: ID does not exist"
Jul 28 16:38:29.209634 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209631    2580 scope.go:117] "RemoveContainer" containerID="159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"
Jul 28 16:38:29.209873 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:38:29.209858    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898\": container with ID starting with 159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898 not found: ID does not exist" containerID="159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"
Jul 28 16:38:29.209920 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.209874    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898"} err="failed to get container status \"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898\": rpc error: code = NotFound desc = could not find container \"159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898\": container with ID starting with 159fb4db56e061359caddcd72803de1432c79bf9e96487e8b8eb001478798898 not found: ID does not exist"
Jul 28 16:38:29.214192 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.214172    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rbfmp"]
Jul 28 16:38:29.299687 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:29.299649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:38:30.187619 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:38:30.187576    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" path="/var/lib/kubelet/pods/2a2c7d38-f524-43ea-a4a8-f7def96db5c9/volumes"
Jul 28 16:39:20.391583 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:39:20.391544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:39:59.299945 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:39:59.299902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:04.389350 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:04.389306    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:05.111008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:05.110974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:06.496369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:06.496316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:11.602391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.602338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:11.629868 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.629835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:40:11.630268 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630252    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kube-rbac-proxy"
Jul 28 16:40:11.630268 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630269    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kube-rbac-proxy"
Jul 28 16:40:11.630378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630283    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container"
Jul 28 16:40:11.630378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630289    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container"
Jul 28 16:40:11.630378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630312    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="storage-initializer"
Jul 28 16:40:11.630378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630318    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="storage-initializer"
Jul 28 16:40:11.630378 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630377    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kserve-container"
Jul 28 16:40:11.630553 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.630388    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2c7d38-f524-43ea-a4a8-f7def96db5c9" containerName="kube-rbac-proxy"
Jul 28 16:40:11.633388 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.633371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.636099 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.636073    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:40:11.636253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.636104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:40:11.636253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.636083    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:40:11.636253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.636183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:40:11.637356 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.637332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:40:11.637568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.637374    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 16:40:11.637568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.637408    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 16:40:11.643819 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.643783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:40:11.685335 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.685306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.685487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.685344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r6g2\" (UniqueName: \"kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.685487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.685375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.685487 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.685429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.685595 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.685506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.786468 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.786434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r6g2\" (UniqueName: \"kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.786655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.786476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.786655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.786522    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.786655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.786566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.786655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.786605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.787045 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.787023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.787318 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.787300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.787362 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.787319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.789299 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.789281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.794813 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.794780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r6g2\" (UniqueName: \"kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2\") pod \"isvc-sklearn-predictor-5f4d89cc44-bzwqz\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:11.830914 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.830881    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:11.944388 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:11.944308    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:12.079854 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:12.079827    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:40:12.082442 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:40:12.082411    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d957403_ddba_4c38_a81c_5797074fad37.slice/crio-d3677634f5ad7bb553ac23340c86e6dadf0f5e8bf423e73f8dd2acdf298fe567 WatchSource:0}: Error finding container d3677634f5ad7bb553ac23340c86e6dadf0f5e8bf423e73f8dd2acdf298fe567: Status 404 returned error can't find the container with id d3677634f5ad7bb553ac23340c86e6dadf0f5e8bf423e73f8dd2acdf298fe567
Jul 28 16:40:12.620637 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:12.620602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerStarted","Data":"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"}
Jul 28 16:40:12.621191 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:12.620664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerStarted","Data":"d3677634f5ad7bb553ac23340c86e6dadf0f5e8bf423e73f8dd2acdf298fe567"}
Jul 28 16:40:16.639284 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:16.639253    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d957403-ddba-4c38-a81c-5797074fad37" containerID="8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d" exitCode=0
Jul 28 16:40:16.639714 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:16.639334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerDied","Data":"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"}
Jul 28 16:40:17.646584 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.646548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerStarted","Data":"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"}
Jul 28 16:40:17.646981 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.646620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerStarted","Data":"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"}
Jul 28 16:40:17.646981 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.646903    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:17.647071 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.647055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:17.648307 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.648282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:40:17.669521 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:17.669466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podStartSLOduration=2.747924487 podStartE2EDuration="6.669448903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.921524416s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:40:17.668174874 +0000 UTC m=+4032.036827790" watchObservedRunningTime="2026-07-28 16:40:17.669448903 +0000 UTC m=+4032.038101818"
Jul 28 16:40:18.650731 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:18.650688    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:40:23.655306 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:23.655276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:40:23.655809 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:23.655780    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:40:33.655899 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:33.655859    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:40:43.656099 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:43.656052    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:40:43.984403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:43.984295    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:51.017337 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:51.017292    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:40:53.655677 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:40:53.655641    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:41:03.656300 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:03.656257    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:41:13.656063 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:13.656015    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:41:23.656601 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:23.656556    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:41:32.495387 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:32.495338    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:41:33.657203 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:33.657175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:41:41.806880 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:41.806847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:41:41.807302 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:41.807166    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" containerID="cri-o://061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30" gracePeriod=30
Jul 28 16:41:41.807302 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:41.807197    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kube-rbac-proxy" containerID="cri-o://a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309" gracePeriod=30
Jul 28 16:41:41.830623 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:41.830588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:41:42.019508 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:42.019470    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d957403-ddba-4c38-a81c-5797074fad37" containerID="a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309" exitCode=2
Jul 28 16:41:42.019668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:42.019529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerDied","Data":"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"}
Jul 28 16:41:43.651456 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:43.651416    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 28 16:41:43.655735 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:43.655694    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 16:41:46.656158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.656112    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:41:46.753567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location\") pod \"3d957403-ddba-4c38-a81c-5797074fad37\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") "
Jul 28 16:41:46.753567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert\") pod \"3d957403-ddba-4c38-a81c-5797074fad37\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") "
Jul 28 16:41:46.753567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753548    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r6g2\" (UniqueName: \"kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2\") pod \"3d957403-ddba-4c38-a81c-5797074fad37\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") "
Jul 28 16:41:46.753890 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753614    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls\") pod \"3d957403-ddba-4c38-a81c-5797074fad37\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") "
Jul 28 16:41:46.753890 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config\") pod \"3d957403-ddba-4c38-a81c-5797074fad37\" (UID: \"3d957403-ddba-4c38-a81c-5797074fad37\") "
Jul 28 16:41:46.753890 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d957403-ddba-4c38-a81c-5797074fad37" (UID: "3d957403-ddba-4c38-a81c-5797074fad37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:41:46.754041 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.753929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d957403-ddba-4c38-a81c-5797074fad37" (UID: "3d957403-ddba-4c38-a81c-5797074fad37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:41:46.754178 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.754151    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d957403-ddba-4c38-a81c-5797074fad37" (UID: "3d957403-ddba-4c38-a81c-5797074fad37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:41:46.755870 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.755847    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d957403-ddba-4c38-a81c-5797074fad37" (UID: "3d957403-ddba-4c38-a81c-5797074fad37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:41:46.755967 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.755867    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2" (OuterVolumeSpecName: "kube-api-access-2r6g2") pod "3d957403-ddba-4c38-a81c-5797074fad37" (UID: "3d957403-ddba-4c38-a81c-5797074fad37"). InnerVolumeSpecName "kube-api-access-2r6g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:41:46.855194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.855161    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:41:46.855194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.855185    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d957403-ddba-4c38-a81c-5797074fad37-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:41:46.855194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.855196    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d957403-ddba-4c38-a81c-5797074fad37-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:41:46.855194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.855205    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r6g2\" (UniqueName: \"kubernetes.io/projected/3d957403-ddba-4c38-a81c-5797074fad37-kube-api-access-2r6g2\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:41:46.855501 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.855214    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d957403-ddba-4c38-a81c-5797074fad37-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:41:46.932114 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.932075    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:41:46.994403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994371    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:41:46.994718 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994705    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="storage-initializer"
Jul 28 16:41:46.994771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994721    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="storage-initializer"
Jul 28 16:41:46.994771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994733    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kube-rbac-proxy"
Jul 28 16:41:46.994771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994742    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kube-rbac-proxy"
Jul 28 16:41:46.994771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994767    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container"
Jul 28 16:41:46.994923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994777    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container"
Jul 28 16:41:46.994923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994861    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kube-rbac-proxy"
Jul 28 16:41:46.994923 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.994883    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d957403-ddba-4c38-a81c-5797074fad37" containerName="kserve-container"
Jul 28 16:41:46.998310 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:46.998293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.000943 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.000916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 16:41:47.001062 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.000950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 16:41:47.012935 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.012910    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:41:47.042701 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.042666    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d957403-ddba-4c38-a81c-5797074fad37" containerID="061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30" exitCode=0
Jul 28 16:41:47.042876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.042730    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"
Jul 28 16:41:47.042876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.042750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerDied","Data":"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"}
Jul 28 16:41:47.042876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.042811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz" event={"ID":"3d957403-ddba-4c38-a81c-5797074fad37","Type":"ContainerDied","Data":"d3677634f5ad7bb553ac23340c86e6dadf0f5e8bf423e73f8dd2acdf298fe567"}
Jul 28 16:41:47.042876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.042838    2580 scope.go:117] "RemoveContainer" containerID="a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"
Jul 28 16:41:47.051617 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.051602    2580 scope.go:117] "RemoveContainer" containerID="061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"
Jul 28 16:41:47.057286 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.057260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.057422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.057291    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.057422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.057313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.057422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.057330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.057422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.057388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmdd\" (UniqueName: \"kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.059365 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.059349    2580 scope.go:117] "RemoveContainer" containerID="8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"
Jul 28 16:41:47.067603 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.067583    2580 scope.go:117] "RemoveContainer" containerID="a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"
Jul 28 16:41:47.067896 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:41:47.067876    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309\": container with ID starting with a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309 not found: ID does not exist" containerID="a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"
Jul 28 16:41:47.067968 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.067904    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309"} err="failed to get container status \"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309\": rpc error: code = NotFound desc = could not find container \"a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309\": container with ID starting with a66d3c6fdd061a29e089c097e41605ca4877b2424e9c5b3eef030867e67cc309 not found: ID does not exist"
Jul 28 16:41:47.067968 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.067922    2580 scope.go:117] "RemoveContainer" containerID="061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"
Jul 28 16:41:47.068263 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:41:47.068246    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30\": container with ID starting with 061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30 not found: ID does not exist" containerID="061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"
Jul 28 16:41:47.068326 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.068267    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30"} err="failed to get container status \"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30\": rpc error: code = NotFound desc = could not find container \"061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30\": container with ID starting with 061ca8ab11d21f8bb735628ce3ac52c86624b80de016afa785f65d81a814da30 not found: ID does not exist"
Jul 28 16:41:47.068326 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.068283    2580 scope.go:117] "RemoveContainer" containerID="8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"
Jul 28 16:41:47.068326 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.068284    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:41:47.068524 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:41:47.068503    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d\": container with ID starting with 8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d not found: ID does not exist" containerID="8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"
Jul 28 16:41:47.068568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.068531    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d"} err="failed to get container status \"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d\": rpc error: code = NotFound desc = could not find container \"8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d\": container with ID starting with 8a84677327d70a9d7b139992d464923a3c051b4f766d9f8303813f2cd07e221d not found: ID does not exist"
Jul 28 16:41:47.073060 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.073035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-bzwqz"]
Jul 28 16:41:47.136285 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.136248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:41:47.158340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.158340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.158568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.158568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.158568 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmdd\" (UniqueName: \"kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.158790 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.158766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.159200 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.159181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.159246 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.159179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.161063 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.161045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.169560 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.169539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmdd\" (UniqueName: \"kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.309192 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.309101    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:47.456198 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:47.456166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:41:47.458389 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:41:47.458359    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b00d6f7_89ac_496f_8d0c_4dcff0556a18.slice/crio-65126183ae8dce484e3f38d43ab94698ab4a104ebf591b66338f5b48ffe9cf14 WatchSource:0}: Error finding container 65126183ae8dce484e3f38d43ab94698ab4a104ebf591b66338f5b48ffe9cf14: Status 404 returned error can't find the container with id 65126183ae8dce484e3f38d43ab94698ab4a104ebf591b66338f5b48ffe9cf14
Jul 28 16:41:48.049586 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:48.049549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerStarted","Data":"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"}
Jul 28 16:41:48.050014 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:48.049619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerStarted","Data":"65126183ae8dce484e3f38d43ab94698ab4a104ebf591b66338f5b48ffe9cf14"}
Jul 28 16:41:48.186036 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:48.186004    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d957403-ddba-4c38-a81c-5797074fad37" path="/var/lib/kubelet/pods/3d957403-ddba-4c38-a81c-5797074fad37/volumes"
Jul 28 16:41:52.070578 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:52.070541    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerID="86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7" exitCode=0
Jul 28 16:41:52.070969 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:52.070618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerDied","Data":"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"}
Jul 28 16:41:53.076591 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:53.076559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerStarted","Data":"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"}
Jul 28 16:41:53.076957 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:53.076621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerStarted","Data":"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"}
Jul 28 16:41:53.076957 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:53.076837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:53.077047 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:53.076976    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:41:53.103424 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:53.103367    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podStartSLOduration=3.448283958 podStartE2EDuration="7.103353265s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.655069307s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:41:53.101263401 +0000 UTC m=+4127.469916322" watchObservedRunningTime="2026-07-28 16:41:53.103353265 +0000 UTC m=+4127.472006181"
Jul 28 16:41:59.086742 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:41:59.086662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:42:29.164685 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:29.164630    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 16:42:39.089398 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:39.089367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:42:47.075236 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.075201    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:42:47.075622 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.075456    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container" containerID="cri-o://37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f" gracePeriod=30
Jul 28 16:42:47.075622 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.075524    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy" containerID="cri-o://17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6" gracePeriod=30
Jul 28 16:42:47.117974 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.117939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:42:47.314778 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.314745    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerID="17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6" exitCode=2
Jul 28 16:42:47.314937 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:47.314833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerDied","Data":"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"}
Jul 28 16:42:49.082363 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:49.082323    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 16:42:50.128303 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:50.128256    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.56:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 16:42:52.211225 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.211177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:42:52.239651 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.239617    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:42:52.243384 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.243366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.246174 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.246150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:42:52.246293 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.246188    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 16:42:52.254718 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.254695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:42:52.331831 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.331785    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:42:52.348474 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.348444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.348474 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.348475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.348658 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.348517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.348658 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.348571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.348658 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.348618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9vs4\" (UniqueName: \"kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.449505 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.449468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.449705 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.449538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.449705 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.449582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.449705 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.449650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9vs4\" (UniqueName: \"kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.449705 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.449689    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.450086 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.450067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.450312 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.450289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.450312 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.450302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.452380 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.452363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.459343 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.459320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9vs4\" (UniqueName: \"kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-48hxr\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.554047 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.553964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:42:52.710022 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:52.709994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:42:52.713504 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:42:52.713472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4cb00c_54f9_4212_9d4c_2b3851a61b5a.slice/crio-8a94f128b5f7c58f09b2981c6b30005124bee877487105fe2d10b73e83cabb38 WatchSource:0}: Error finding container 8a94f128b5f7c58f09b2981c6b30005124bee877487105fe2d10b73e83cabb38: Status 404 returned error can't find the container with id 8a94f128b5f7c58f09b2981c6b30005124bee877487105fe2d10b73e83cabb38
Jul 28 16:42:53.342714 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:53.342675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerStarted","Data":"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"}
Jul 28 16:42:53.343064 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:53.342743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerStarted","Data":"8a94f128b5f7c58f09b2981c6b30005124bee877487105fe2d10b73e83cabb38"}
Jul 28 16:42:54.082319 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.082269    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 16:42:54.725474 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.725450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:42:54.871171 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location\") pod \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") "
Jul 28 16:42:54.871336 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert\") pod \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") "
Jul 28 16:42:54.871336 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls\") pod \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") "
Jul 28 16:42:54.871336 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config\") pod \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") "
Jul 28 16:42:54.871510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqmdd\" (UniqueName: \"kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd\") pod \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\" (UID: \"4b00d6f7-89ac-496f-8d0c-4dcff0556a18\") "
Jul 28 16:42:54.871664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b00d6f7-89ac-496f-8d0c-4dcff0556a18" (UID: "4b00d6f7-89ac-496f-8d0c-4dcff0556a18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:42:54.871824 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b00d6f7-89ac-496f-8d0c-4dcff0556a18" (UID: "4b00d6f7-89ac-496f-8d0c-4dcff0556a18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:54.871883 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.871850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b00d6f7-89ac-496f-8d0c-4dcff0556a18" (UID: "4b00d6f7-89ac-496f-8d0c-4dcff0556a18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:54.873420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.873398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b00d6f7-89ac-496f-8d0c-4dcff0556a18" (UID: "4b00d6f7-89ac-496f-8d0c-4dcff0556a18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:42:54.873765 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.873732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd" (OuterVolumeSpecName: "kube-api-access-jqmdd") pod "4b00d6f7-89ac-496f-8d0c-4dcff0556a18" (UID: "4b00d6f7-89ac-496f-8d0c-4dcff0556a18"). InnerVolumeSpecName "kube-api-access-jqmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:42:54.972862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.972829    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:42:54.972862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.972855    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:42:54.972862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.972864    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:42:54.972862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.972872    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmdd\" (UniqueName: \"kubernetes.io/projected/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kube-api-access-jqmdd\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:42:54.973149 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:54.972880    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b00d6f7-89ac-496f-8d0c-4dcff0556a18-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:42:55.352967 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.352932    2580 generic.go:356] "Generic (PLEG): container finished" podID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerID="37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f" exitCode=0
Jul 28 16:42:55.353149 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.353004    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"
Jul 28 16:42:55.353149 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.353013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerDied","Data":"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"}
Jul 28 16:42:55.353149 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.353114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd" event={"ID":"4b00d6f7-89ac-496f-8d0c-4dcff0556a18","Type":"ContainerDied","Data":"65126183ae8dce484e3f38d43ab94698ab4a104ebf591b66338f5b48ffe9cf14"}
Jul 28 16:42:55.353270 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.353157    2580 scope.go:117] "RemoveContainer" containerID="17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"
Jul 28 16:42:55.362673 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.362656    2580 scope.go:117] "RemoveContainer" containerID="37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"
Jul 28 16:42:55.370668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.370652    2580 scope.go:117] "RemoveContainer" containerID="86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"
Jul 28 16:42:55.377966 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.377902    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:42:55.379720 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.379703    2580 scope.go:117] "RemoveContainer" containerID="17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"
Jul 28 16:42:55.380056 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:42:55.380036    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6\": container with ID starting with 17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6 not found: ID does not exist" containerID="17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"
Jul 28 16:42:55.380112 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380068    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6"} err="failed to get container status \"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6\": rpc error: code = NotFound desc = could not find container \"17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6\": container with ID starting with 17d47b14c591644161e4731e73e59354f3124775b54cae44451e16569e4a11b6 not found: ID does not exist"
Jul 28 16:42:55.380112 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380086    2580 scope.go:117] "RemoveContainer" containerID="37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"
Jul 28 16:42:55.380403 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:42:55.380381    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f\": container with ID starting with 37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f not found: ID does not exist" containerID="37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"
Jul 28 16:42:55.380472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380414    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cqnvd"]
Jul 28 16:42:55.380472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380411    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f"} err="failed to get container status \"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f\": rpc error: code = NotFound desc = could not find container \"37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f\": container with ID starting with 37ed5df788f189f4e65809aa5337ae712c458b719aeba937d46da2455556f47f not found: ID does not exist"
Jul 28 16:42:55.380472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380431    2580 scope.go:117] "RemoveContainer" containerID="86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"
Jul 28 16:42:55.380680 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:42:55.380660    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7\": container with ID starting with 86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7 not found: ID does not exist" containerID="86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"
Jul 28 16:42:55.380746 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:55.380684    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7"} err="failed to get container status \"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7\": rpc error: code = NotFound desc = could not find container \"86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7\": container with ID starting with 86283a611bd38e2c0bdfc0f9964f08a688448ec2a2b257f8a5bb86ba1f0931e7 not found: ID does not exist"
Jul 28 16:42:56.187793 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:56.187759    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" path="/var/lib/kubelet/pods/4b00d6f7-89ac-496f-8d0c-4dcff0556a18/volumes"
Jul 28 16:42:59.373073 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:59.373039    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerID="67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6" exitCode=0
Jul 28 16:42:59.373466 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:42:59.373113    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerDied","Data":"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"}
Jul 28 16:43:00.387274 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:00.387239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerStarted","Data":"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"}
Jul 28 16:43:00.387674 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:00.387335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerStarted","Data":"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"}
Jul 28 16:43:00.387674 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:00.387650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:00.409452 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:00.409395    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podStartSLOduration=2.543310817 podStartE2EDuration="8.409377229s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.866066412s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:43:00.408211124 +0000 UTC m=+4194.776864041" watchObservedRunningTime="2026-07-28 16:43:00.409377229 +0000 UTC m=+4194.778030144"
Jul 28 16:43:01.393015 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:01.392983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:01.394315 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:01.394282    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 16:43:02.397827 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:02.397782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 16:43:07.402965 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:07.402936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:07.403550 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:07.403520    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 16:43:17.404287 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:17.404253    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:24.205193 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.205157    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-599b58cf6-48hxr_fb4cb00c-54f9-4212-9d4c-2b3851a61b5a/kserve-container/0.log"
Jul 28 16:43:24.368761 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.368727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:43:24.369044 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.369017    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container" containerID="cri-o://faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d" gracePeriod=30
Jul 28 16:43:24.369158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.369056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kube-rbac-proxy" containerID="cri-o://374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de" gracePeriod=30
Jul 28 16:43:24.391755 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.391723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:43:24.510353 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.510323    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerID="374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de" exitCode=2
Jul 28 16:43:24.510490 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:24.510397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerDied","Data":"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"}
Jul 28 16:43:25.322497 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.322474    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:25.468420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location\") pod \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") "
Jul 28 16:43:25.468420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls\") pod \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") "
Jul 28 16:43:25.468420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert\") pod \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") "
Jul 28 16:43:25.468688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9vs4\" (UniqueName: \"kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4\") pod \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") "
Jul 28 16:43:25.468688 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468621    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config\") pod \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\" (UID: \"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a\") "
Jul 28 16:43:25.468796 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" (UID: "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:43:25.468902 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468884    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:43:25.468991 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.468954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" (UID: "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:43:25.471271 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.471241    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" (UID: "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:43:25.471271 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.471262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4" (OuterVolumeSpecName: "kube-api-access-p9vs4") pod "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" (UID: "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a"). InnerVolumeSpecName "kube-api-access-p9vs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:43:25.472340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.472317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" (UID: "fb4cb00c-54f9-4212-9d4c-2b3851a61b5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:43:25.516763 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.516725    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerID="faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d" exitCode=0
Jul 28 16:43:25.516763 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.516768    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerDied","Data":"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"}
Jul 28 16:43:25.516987 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.516794    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"
Jul 28 16:43:25.516987 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.516817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr" event={"ID":"fb4cb00c-54f9-4212-9d4c-2b3851a61b5a","Type":"ContainerDied","Data":"8a94f128b5f7c58f09b2981c6b30005124bee877487105fe2d10b73e83cabb38"}
Jul 28 16:43:25.516987 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.516844    2580 scope.go:117] "RemoveContainer" containerID="374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"
Jul 28 16:43:25.528393 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.527196    2580 scope.go:117] "RemoveContainer" containerID="faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"
Jul 28 16:43:25.536110 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.536087    2580 scope.go:117] "RemoveContainer" containerID="67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"
Jul 28 16:43:25.540759 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.540736    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:43:25.544091 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.544074    2580 scope.go:117] "RemoveContainer" containerID="374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"
Jul 28 16:43:25.544381 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:43:25.544360    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de\": container with ID starting with 374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de not found: ID does not exist" containerID="374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"
Jul 28 16:43:25.544454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.544391    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de"} err="failed to get container status \"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de\": rpc error: code = NotFound desc = could not find container \"374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de\": container with ID starting with 374bc01f116ed05ea585235ebe5ad7d50d30e20be2a38961bca2537c1594c7de not found: ID does not exist"
Jul 28 16:43:25.544454 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.544410    2580 scope.go:117] "RemoveContainer" containerID="faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"
Jul 28 16:43:25.544657 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:43:25.544631    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d\": container with ID starting with faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d not found: ID does not exist" containerID="faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"
Jul 28 16:43:25.544708 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.544667    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d"} err="failed to get container status \"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d\": rpc error: code = NotFound desc = could not find container \"faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d\": container with ID starting with faba11c0ea83cd9571558e0e6781014c5b56b45a210a8095cba637e14730e97d not found: ID does not exist"
Jul 28 16:43:25.544708 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.544687    2580 scope.go:117] "RemoveContainer" containerID="67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"
Jul 28 16:43:25.545260 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:43:25.545234    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6\": container with ID starting with 67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6 not found: ID does not exist" containerID="67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"
Jul 28 16:43:25.545370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.545269    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6"} err="failed to get container status \"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6\": rpc error: code = NotFound desc = could not find container \"67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6\": container with ID starting with 67069f98fd21470814d7ec1757ff5b9b2c24608680c673e698768ea5a40522b6 not found: ID does not exist"
Jul 28 16:43:25.547201 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.547181    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-48hxr"]
Jul 28 16:43:25.570340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.570313    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:43:25.570340 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.570338    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9vs4\" (UniqueName: \"kubernetes.io/projected/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-api-access-p9vs4\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:43:25.570470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.570347    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:43:25.570470 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:25.570357    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:43:26.186360 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:26.186262    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" path="/var/lib/kubelet/pods/fb4cb00c-54f9-4212-9d4c-2b3851a61b5a/volumes"
Jul 28 16:43:28.418004 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:28.417969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:43:29.590369 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590693    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590704    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590713    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="storage-initializer"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590718    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="storage-initializer"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590726    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="storage-initializer"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590732    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="storage-initializer"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590746    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590751    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590767    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590774    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kube-rbac-proxy"
Jul 28 16:43:29.590803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590778    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kube-rbac-proxy"
Jul 28 16:43:29.591251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590851    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kserve-container"
Jul 28 16:43:29.591251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590865    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kserve-container"
Jul 28 16:43:29.591251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590875    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4cb00c-54f9-4212-9d4c-2b3851a61b5a" containerName="kube-rbac-proxy"
Jul 28 16:43:29.591251 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.590884    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b00d6f7-89ac-496f-8d0c-4dcff0556a18" containerName="kube-rbac-proxy"
Jul 28 16:43:29.595779 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.595761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.598694 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.598670    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:43:29.598832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.598672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:43:29.598832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.598713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:43:29.598832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.598720    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:43:29.598832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.598785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:43:29.599798 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.599782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 16:43:29.599885 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.599818    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:43:29.605564 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.605543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:43:29.708167 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.708107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.708303 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.708239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.708303 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.708290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.708395 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.708340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.708395 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.708368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvdj\" (UniqueName: \"kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809032 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.808996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvdj\" (UniqueName: \"kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809209 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.809054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809209 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.809101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809209 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.809163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809209 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.809203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.809633 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.809613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.810028 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.810004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.810120 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.810042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.811837 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.811820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.820017 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.819994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvdj\" (UniqueName: \"kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:29.907066 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:29.906967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:30.046786 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:30.046762    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:43:30.049251 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:43:30.049219    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ea03b5_c07f_4054_8561_38e3ba7654bb.slice/crio-c59b6d864208847de525882cbf2a18f7ccb785b9014d2b8429d8456b035851cf WatchSource:0}: Error finding container c59b6d864208847de525882cbf2a18f7ccb785b9014d2b8429d8456b035851cf: Status 404 returned error can't find the container with id c59b6d864208847de525882cbf2a18f7ccb785b9014d2b8429d8456b035851cf
Jul 28 16:43:30.226514 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:30.226400    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:43:30.540522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:30.540423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerStarted","Data":"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"}
Jul 28 16:43:30.540522 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:30.540480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerStarted","Data":"c59b6d864208847de525882cbf2a18f7ccb785b9014d2b8429d8456b035851cf"}
Jul 28 16:43:34.559862 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:34.559830    2580 generic.go:356] "Generic (PLEG): container finished" podID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerID="f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74" exitCode=0
Jul 28 16:43:34.560412 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:34.559905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerDied","Data":"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"}
Jul 28 16:43:35.567288 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:35.567251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerStarted","Data":"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"}
Jul 28 16:43:35.567726 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:35.567315    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerStarted","Data":"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"}
Jul 28 16:43:35.567726 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:35.567505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:35.588930 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:35.588875    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" podStartSLOduration=2.900948433 podStartE2EDuration="6.588861047s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687912614s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:43:35.588045729 +0000 UTC m=+4229.956698647" watchObservedRunningTime="2026-07-28 16:43:35.588861047 +0000 UTC m=+4229.957514053"
Jul 28 16:43:36.572653 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:36.572626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:43:42.583224 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:43:42.583185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:44:12.664632 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:12.664584    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 16:44:22.586236 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:22.586205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:44:29.638088 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.638057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:44:29.638530 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.638344    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kserve-container" containerID="cri-o://800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947" gracePeriod=30
Jul 28 16:44:29.638530 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.638384    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kube-rbac-proxy" containerID="cri-o://70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4" gracePeriod=30
Jul 28 16:44:29.663086 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.663053    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:44:29.812289 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.812252    2580 generic.go:356] "Generic (PLEG): container finished" podID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerID="70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4" exitCode=2
Jul 28 16:44:29.812472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:29.812322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerDied","Data":"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"}
Jul 28 16:44:32.577929 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:32.577880    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 28 16:44:34.791272 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.791236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:44:34.822413 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.822379    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:44:34.826183 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.826159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.829038 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.829012    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 16:44:34.829192 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.829012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 16:44:34.836876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.836849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:44:34.883462 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.883435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.883611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.883475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.883611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.883549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.883611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.883598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmbz\" (UniqueName: \"kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.883728 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.883626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.984887 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.984846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985061 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.984909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985061 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.984991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985061 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:44:34.985013    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 16:44:34.985061 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.985020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmbz\" (UniqueName: \"kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985061 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.985045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985321 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:44:34.985086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls podName:c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b nodeName:}" failed. No retries permitted until 2026-07-28 16:44:35.485063907 +0000 UTC m=+4289.853716821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls") pod "isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 16:44:34.985443 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.985425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.985648    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.985719 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.985698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:34.994278 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:34.994258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmbz\" (UniqueName: \"kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:35.202819 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:35.202786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:44:35.491827 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:35.491736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:35.494333 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:35.494312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:35.738749 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:35.738706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:35.879592 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:35.879571    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:44:35.881830 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:44:35.881804    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e99feb_c12e_4ef9_9c96_375e9fc3fd1b.slice/crio-fd017bb6b8460e0a14a6bb4e19d12ed2e3bff04303ab5509340ccc7aebb57c4e WatchSource:0}: Error finding container fd017bb6b8460e0a14a6bb4e19d12ed2e3bff04303ab5509340ccc7aebb57c4e: Status 404 returned error can't find the container with id fd017bb6b8460e0a14a6bb4e19d12ed2e3bff04303ab5509340ccc7aebb57c4e
Jul 28 16:44:36.845757 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:36.845721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerStarted","Data":"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"}
Jul 28 16:44:36.846001 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:36.845779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerStarted","Data":"fd017bb6b8460e0a14a6bb4e19d12ed2e3bff04303ab5509340ccc7aebb57c4e"}
Jul 28 16:44:37.197186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.197162    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:44:37.306219 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location\") pod \"85ea03b5-c07f-4054-8561-38e3ba7654bb\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") "
Jul 28 16:44:37.306423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls\") pod \"85ea03b5-c07f-4054-8561-38e3ba7654bb\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") "
Jul 28 16:44:37.306423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config\") pod \"85ea03b5-c07f-4054-8561-38e3ba7654bb\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") "
Jul 28 16:44:37.306423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306364    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert\") pod \"85ea03b5-c07f-4054-8561-38e3ba7654bb\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") "
Jul 28 16:44:37.306423 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306415    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcvdj\" (UniqueName: \"kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj\") pod \"85ea03b5-c07f-4054-8561-38e3ba7654bb\" (UID: \"85ea03b5-c07f-4054-8561-38e3ba7654bb\") "
Jul 28 16:44:37.306676 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85ea03b5-c07f-4054-8561-38e3ba7654bb" (UID: "85ea03b5-c07f-4054-8561-38e3ba7654bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:44:37.306727 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85ea03b5-c07f-4054-8561-38e3ba7654bb" (UID: "85ea03b5-c07f-4054-8561-38e3ba7654bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:37.306727 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.306703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85ea03b5-c07f-4054-8561-38e3ba7654bb" (UID: "85ea03b5-c07f-4054-8561-38e3ba7654bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:37.308557 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.308537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85ea03b5-c07f-4054-8561-38e3ba7654bb" (UID: "85ea03b5-c07f-4054-8561-38e3ba7654bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:44:37.308621 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.308598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj" (OuterVolumeSpecName: "kube-api-access-bcvdj") pod "85ea03b5-c07f-4054-8561-38e3ba7654bb" (UID: "85ea03b5-c07f-4054-8561-38e3ba7654bb"). InnerVolumeSpecName "kube-api-access-bcvdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:44:37.408008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.407913    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ea03b5-c07f-4054-8561-38e3ba7654bb-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:44:37.408008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.407944    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ea03b5-c07f-4054-8561-38e3ba7654bb-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:44:37.408008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.407954    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:44:37.408008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.407962    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ea03b5-c07f-4054-8561-38e3ba7654bb-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:44:37.408008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.407971    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcvdj\" (UniqueName: \"kubernetes.io/projected/85ea03b5-c07f-4054-8561-38e3ba7654bb-kube-api-access-bcvdj\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:44:37.852212 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.852179    2580 generic.go:356] "Generic (PLEG): container finished" podID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerID="800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947" exitCode=0
Jul 28 16:44:37.852459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.852246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"
Jul 28 16:44:37.852459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.852267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerDied","Data":"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"}
Jul 28 16:44:37.852459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.852336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt" event={"ID":"85ea03b5-c07f-4054-8561-38e3ba7654bb","Type":"ContainerDied","Data":"c59b6d864208847de525882cbf2a18f7ccb785b9014d2b8429d8456b035851cf"}
Jul 28 16:44:37.852459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.852373    2580 scope.go:117] "RemoveContainer" containerID="70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"
Jul 28 16:44:37.861605 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.861588    2580 scope.go:117] "RemoveContainer" containerID="800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"
Jul 28 16:44:37.869630 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.869609    2580 scope.go:117] "RemoveContainer" containerID="f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"
Jul 28 16:44:37.875563 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.875539    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:44:37.878292 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.878273    2580 scope.go:117] "RemoveContainer" containerID="70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"
Jul 28 16:44:37.878591 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:44:37.878568    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4\": container with ID starting with 70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4 not found: ID does not exist" containerID="70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"
Jul 28 16:44:37.878686 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.878598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4"} err="failed to get container status \"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4\": rpc error: code = NotFound desc = could not find container \"70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4\": container with ID starting with 70b0d32ee243549c1cde480d759a1f2e8362bba57bd9bc590c2936eb2beb26e4 not found: ID does not exist"
Jul 28 16:44:37.878686 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.878617    2580 scope.go:117] "RemoveContainer" containerID="800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"
Jul 28 16:44:37.878917 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:44:37.878895    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947\": container with ID starting with 800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947 not found: ID does not exist" containerID="800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"
Jul 28 16:44:37.878958 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.878926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947"} err="failed to get container status \"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947\": rpc error: code = NotFound desc = could not find container \"800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947\": container with ID starting with 800e18db43a7f64277507507b3e099a7f1a773cd51e88fec8045774359923947 not found: ID does not exist"
Jul 28 16:44:37.878958 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.878949    2580 scope.go:117] "RemoveContainer" containerID="f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"
Jul 28 16:44:37.879269 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:44:37.879247    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74\": container with ID starting with f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74 not found: ID does not exist" containerID="f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"
Jul 28 16:44:37.879349 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.879277    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74"} err="failed to get container status \"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74\": rpc error: code = NotFound desc = could not find container \"f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74\": container with ID starting with f714f94510433900074ab6c93b3c5554b62ed0ce386775e4fe179518fd570f74 not found: ID does not exist"
Jul 28 16:44:37.879851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:37.879834    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-wgzqt"]
Jul 28 16:44:38.186317 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:38.186239    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" path="/var/lib/kubelet/pods/85ea03b5-c07f-4054-8561-38e3ba7654bb/volumes"
Jul 28 16:44:40.870321 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:40.870283    2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerID="baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305" exitCode=0
Jul 28 16:44:40.870778 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:40.870362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerDied","Data":"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"}
Jul 28 16:44:41.878457 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:41.878421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerStarted","Data":"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"}
Jul 28 16:44:41.878896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:41.878484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerStarted","Data":"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"}
Jul 28 16:44:41.878896 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:41.878673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:41.899689 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:41.899630    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podStartSLOduration=3.9267760640000002 podStartE2EDuration="7.899615195s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972839131s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:44:41.89776951 +0000 UTC m=+4296.266422425" watchObservedRunningTime="2026-07-28 16:44:41.899615195 +0000 UTC m=+4296.268268111"
Jul 28 16:44:42.883859 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:42.883824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:42.885119 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:42.885087    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:44:43.888812 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:43.888769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:44:48.893279 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:48.893247    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:44:48.893909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:48.893883    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:44:58.894408 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:44:58.894321    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:08.894409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:08.894361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:18.894704 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:18.894664    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:28.894832 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:28.894790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:38.893852 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:38.893805    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:48.894580 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:48.894532    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:45:58.895307 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:45:58.895267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:46:05.079194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.079154    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:46:05.079622 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.079370    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" containerID="cri-o://d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d" gracePeriod=30
Jul 28 16:46:05.079622 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.079422    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kube-rbac-proxy" containerID="cri-o://165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61" gracePeriod=30
Jul 28 16:46:05.104503 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.104472    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:46:05.241227 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.241193    2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerID="165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61" exitCode=2
Jul 28 16:46:05.241392 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:05.241274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerDied","Data":"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"}
Jul 28 16:46:08.889911 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:08.889860    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jul 28 16:46:08.894767 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:08.894738    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 16:46:09.928114 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:09.928087    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:46:10.020950 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.020869    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmmbz\" (UniqueName: \"kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz\") pod \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") "
Jul 28 16:46:10.020950 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.020907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location\") pod \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") "
Jul 28 16:46:10.021186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.020952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") pod \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") "
Jul 28 16:46:10.021186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.021036    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config\") pod \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") "
Jul 28 16:46:10.021186 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.021064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert\") pod \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\" (UID: \"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b\") "
Jul 28 16:46:10.021415 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.021377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:46:10.021536 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.021482    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:46:10.021536 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.021483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:46:10.023191 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.023166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:46:10.023283 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.023259    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz" (OuterVolumeSpecName: "kube-api-access-wmmbz") pod "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" (UID: "c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b"). InnerVolumeSpecName "kube-api-access-wmmbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:46:10.122096 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.122052    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:46:10.122096 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.122085    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:46:10.122096 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.122098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmmbz\" (UniqueName: \"kubernetes.io/projected/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kube-api-access-wmmbz\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:46:10.122366 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.122110    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:46:10.122366 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.122122    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:46:10.240075 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.240036    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:46:10.266269 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.266230    2580 generic.go:356] "Generic (PLEG): container finished" podID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerID="d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d" exitCode=0
Jul 28 16:46:10.266442 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.266295    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"
Jul 28 16:46:10.266442 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.266317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerDied","Data":"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"}
Jul 28 16:46:10.266442 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.266379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg" event={"ID":"c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b","Type":"ContainerDied","Data":"fd017bb6b8460e0a14a6bb4e19d12ed2e3bff04303ab5509340ccc7aebb57c4e"}
Jul 28 16:46:10.266442 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.266408    2580 scope.go:117] "RemoveContainer" containerID="165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"
Jul 28 16:46:10.274286 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274226    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:46:10.274571 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274558    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="storage-initializer"
Jul 28 16:46:10.274610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274575    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="storage-initializer"
Jul 28 16:46:10.274610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274596    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container"
Jul 28 16:46:10.274610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274602    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container"
Jul 28 16:46:10.274610 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274609    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kube-rbac-proxy"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274615    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kube-rbac-proxy"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274626    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kserve-container"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274631    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kserve-container"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="storage-initializer"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274646    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="storage-initializer"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274654    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kube-rbac-proxy"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274659    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kube-rbac-proxy"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kserve-container"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274716    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" containerName="kube-rbac-proxy"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274723    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kserve-container"
Jul 28 16:46:10.274734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.274730    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ea03b5-c07f-4054-8561-38e3ba7654bb" containerName="kube-rbac-proxy"
Jul 28 16:46:10.275499 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.275480    2580 scope.go:117] "RemoveContainer" containerID="d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"
Jul 28 16:46:10.279165 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.279146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.281966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.281989    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.281973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.281989    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.282034    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.281966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 16:46:10.282152 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.282012    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:46:10.284491 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.284472    2580 scope.go:117] "RemoveContainer" containerID="baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"
Jul 28 16:46:10.290172 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.290145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:46:10.296157 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.296120    2580 scope.go:117] "RemoveContainer" containerID="165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"
Jul 28 16:46:10.296619 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:46:10.296590    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61\": container with ID starting with 165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61 not found: ID does not exist" containerID="165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"
Jul 28 16:46:10.296732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.296627    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61"} err="failed to get container status \"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61\": rpc error: code = NotFound desc = could not find container \"165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61\": container with ID starting with 165a5c2de005f14a6bec76d4972d023772708b3e6248bd06c15c2c3f33bd1b61 not found: ID does not exist"
Jul 28 16:46:10.296732 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.296651    2580 scope.go:117] "RemoveContainer" containerID="d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"
Jul 28 16:46:10.297565 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:46:10.297542    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d\": container with ID starting with d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d not found: ID does not exist" containerID="d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"
Jul 28 16:46:10.297659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.297570    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d"} err="failed to get container status \"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d\": rpc error: code = NotFound desc = could not find container \"d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d\": container with ID starting with d8b9db413894a07acceb1aa2febe6162103159342292b182309d490f7ba1357d not found: ID does not exist"
Jul 28 16:46:10.297659 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.297587    2580 scope.go:117] "RemoveContainer" containerID="baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"
Jul 28 16:46:10.297834 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:46:10.297816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305\": container with ID starting with baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305 not found: ID does not exist" containerID="baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"
Jul 28 16:46:10.297881 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.297840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305"} err="failed to get container status \"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305\": rpc error: code = NotFound desc = could not find container \"baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305\": container with ID starting with baded0228de4f94f002d55ccffa934d94d322bb81c28b67a68bd46204a0e9305 not found: ID does not exist"
Jul 28 16:46:10.298718 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.298700    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:46:10.304582 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.304563    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-h2zhg"]
Jul 28 16:46:10.324221 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.324190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.324341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.324246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.324341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.324324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.324434 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.324377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.324479 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.324455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jk4\" (UniqueName: \"kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.416664 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.416627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:46:10.425712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.425686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.425857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.425730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.425857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.425766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.425857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.425827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.425998 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.425890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jk4\" (UniqueName: \"kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.426237 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.426210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.426561 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.426532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.426655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.426535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.428425 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.428407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.434206 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.434187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jk4\" (UniqueName: \"kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.592053 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.592018    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:10.727802 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:10.727772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:46:10.730557 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:46:10.730521    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df6aa5c_73f6_4f58_b2b4_aaa19846a614.slice/crio-42f6ad0c61afa3120172d6ac1871b28a108560df82011829b483f4d4e6283bce WatchSource:0}: Error finding container 42f6ad0c61afa3120172d6ac1871b28a108560df82011829b483f4d4e6283bce: Status 404 returned error can't find the container with id 42f6ad0c61afa3120172d6ac1871b28a108560df82011829b483f4d4e6283bce
Jul 28 16:46:11.274551 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:11.274513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerStarted","Data":"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"}
Jul 28 16:46:11.274917 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:11.274588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerStarted","Data":"42f6ad0c61afa3120172d6ac1871b28a108560df82011829b483f4d4e6283bce"}
Jul 28 16:46:12.190367 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:12.188091    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b" path="/var/lib/kubelet/pods/c0e99feb-c12e-4ef9-9c96-375e9fc3fd1b/volumes"
Jul 28 16:46:15.294806 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:15.294774    2580 generic.go:356] "Generic (PLEG): container finished" podID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerID="4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12" exitCode=0
Jul 28 16:46:15.295314 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:15.294854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerDied","Data":"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"}
Jul 28 16:46:16.301771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:16.301734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerStarted","Data":"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"}
Jul 28 16:46:16.302178 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:16.301795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerStarted","Data":"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"}
Jul 28 16:46:16.302178 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:16.301987    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:16.325075 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:16.325015    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podStartSLOduration=2.57480251 podStartE2EDuration="6.324999645s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750197135s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:46:16.323523166 +0000 UTC m=+4390.692176093" watchObservedRunningTime="2026-07-28 16:46:16.324999645 +0000 UTC m=+4390.693652561"
Jul 28 16:46:17.307527 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:17.307493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:17.308853 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:17.308823    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:46:18.312257 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:18.312208    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:46:23.316593 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:23.316561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:46:23.317221 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:23.317191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:46:33.317371 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:33.317273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:46:43.317377 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:43.317332    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:46:53.317909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:46:53.317862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:47:03.317148 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:03.317084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:47:11.728081 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:11.728039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:47:13.317401 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:13.317360    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:47:23.318107 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:23.318060    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:47:33.317895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:33.317865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:47:40.462533 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.462500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:47:40.463010 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.462721    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" containerID="cri-o://8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c" gracePeriod=30
Jul 28 16:47:40.463010 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.462794    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kube-rbac-proxy" containerID="cri-o://12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416" gracePeriod=30
Jul 28 16:47:40.486299 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.486272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:47:40.686250 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.686213    2580 generic.go:356] "Generic (PLEG): container finished" podID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerID="12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416" exitCode=2
Jul 28 16:47:40.686424 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:40.686285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerDied","Data":"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"}
Jul 28 16:47:43.312704 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:43.312659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Jul 28 16:47:43.318093 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:43.318061    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 16:47:45.591667 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.591628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:47:45.618963 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.618932    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:47:45.656049 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.655926    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:47:45.656315 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656301    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container"
Jul 28 16:47:45.656364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container"
Jul 28 16:47:45.656364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="storage-initializer"
Jul 28 16:47:45.656364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656333    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="storage-initializer"
Jul 28 16:47:45.656364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kube-rbac-proxy"
Jul 28 16:47:45.656364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kube-rbac-proxy"
Jul 28 16:47:45.656545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656404    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kserve-container"
Jul 28 16:47:45.656545 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.656413    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerName="kube-rbac-proxy"
Jul 28 16:47:45.659823 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.659798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.662427 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.662395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 16:47:45.662614 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.662445    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 16:47:45.669714 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.669680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:47:45.680734 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.680704    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4jk4\" (UniqueName: \"kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4\") pod \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") "
Jul 28 16:47:45.681008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.680787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls\") pod \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") "
Jul 28 16:47:45.681008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.680833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location\") pod \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") "
Jul 28 16:47:45.681008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.680874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert\") pod \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") "
Jul 28 16:47:45.681008 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.680933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config\") pod \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\" (UID: \"3df6aa5c-73f6-4f58-b2b4-aaa19846a614\") "
Jul 28 16:47:45.681524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.681496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3df6aa5c-73f6-4f58-b2b4-aaa19846a614" (UID: "3df6aa5c-73f6-4f58-b2b4-aaa19846a614"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:47:45.681524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.681512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3df6aa5c-73f6-4f58-b2b4-aaa19846a614" (UID: "3df6aa5c-73f6-4f58-b2b4-aaa19846a614"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:45.681693 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.681545    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3df6aa5c-73f6-4f58-b2b4-aaa19846a614" (UID: "3df6aa5c-73f6-4f58-b2b4-aaa19846a614"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:45.683689 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.683644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4" (OuterVolumeSpecName: "kube-api-access-f4jk4") pod "3df6aa5c-73f6-4f58-b2b4-aaa19846a614" (UID: "3df6aa5c-73f6-4f58-b2b4-aaa19846a614"). InnerVolumeSpecName "kube-api-access-f4jk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:47:45.683803 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.683702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3df6aa5c-73f6-4f58-b2b4-aaa19846a614" (UID: "3df6aa5c-73f6-4f58-b2b4-aaa19846a614"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:47:45.710964 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.710923    2580 generic.go:356] "Generic (PLEG): container finished" podID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" containerID="8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c" exitCode=0
Jul 28 16:47:45.711158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.711002    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"
Jul 28 16:47:45.711158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.711003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerDied","Data":"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"}
Jul 28 16:47:45.711158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.711070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk" event={"ID":"3df6aa5c-73f6-4f58-b2b4-aaa19846a614","Type":"ContainerDied","Data":"42f6ad0c61afa3120172d6ac1871b28a108560df82011829b483f4d4e6283bce"}
Jul 28 16:47:45.711158 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.711098    2580 scope.go:117] "RemoveContainer" containerID="12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"
Jul 28 16:47:45.722935 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.722908    2580 scope.go:117] "RemoveContainer" containerID="8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"
Jul 28 16:47:45.733139 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.733095    2580 scope.go:117] "RemoveContainer" containerID="4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"
Jul 28 16:47:45.742877 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.742852    2580 scope.go:117] "RemoveContainer" containerID="12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"
Jul 28 16:47:45.743322 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:47:45.743287    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416\": container with ID starting with 12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416 not found: ID does not exist" containerID="12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"
Jul 28 16:47:45.743431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.743322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416"} err="failed to get container status \"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416\": rpc error: code = NotFound desc = could not find container \"12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416\": container with ID starting with 12ec333de31d0cee75bf9771634be720abf07985c1c9141ef8549017a141e416 not found: ID does not exist"
Jul 28 16:47:45.743431 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.743345    2580 scope.go:117] "RemoveContainer" containerID="8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"
Jul 28 16:47:45.743697 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:47:45.743667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c\": container with ID starting with 8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c not found: ID does not exist" containerID="8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"
Jul 28 16:47:45.743756 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.743710    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c"} err="failed to get container status \"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c\": rpc error: code = NotFound desc = could not find container \"8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c\": container with ID starting with 8d56eaea592641423e0d40ddc3dec1f3fe3a37083449677dcb0954ac968e859c not found: ID does not exist"
Jul 28 16:47:45.743756 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.743724    2580 scope.go:117] "RemoveContainer" containerID="4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"
Jul 28 16:47:45.743968 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:47:45.743945    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12\": container with ID starting with 4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12 not found: ID does not exist" containerID="4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"
Jul 28 16:47:45.744015 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.743973    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12"} err="failed to get container status \"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12\": rpc error: code = NotFound desc = could not find container \"4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12\": container with ID starting with 4fae6ef4a8c54598849f4000f10c169ed956437fb139b49bbac9cb14d2d35d12 not found: ID does not exist"
Jul 28 16:47:45.744766 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.744741    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:47:45.748830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.748776    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-wdjdk"]
Jul 28 16:47:45.782418 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.782597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.782597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.782597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.782597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhhk\" (UniqueName: \"kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.782597 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782598    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:47:45.782772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782613    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jk4\" (UniqueName: \"kubernetes.io/projected/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kube-api-access-f4jk4\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:47:45.782772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782624    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:47:45.782772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782635    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:47:45.782772 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.782643    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df6aa5c-73f6-4f58-b2b4-aaa19846a614-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:47:45.883927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.883880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.883927 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.883927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884197 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:47:45.884035    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 28 16:47:45.884197 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884197 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:47:45.884092    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls podName:1903e8ba-4244-46a7-9b76-bf826fea6f57 nodeName:}" failed. No retries permitted until 2026-07-28 16:47:46.384072154 +0000 UTC m=+4480.752725049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-jmhzp" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57") : secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 28 16:47:45.884197 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884197 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhhk\" (UniqueName: \"kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884424 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884759 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.884831 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.884812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.896384 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.896342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhhk\" (UniqueName: \"kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:45.980430 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:45.980333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:47:46.186858 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:46.186821    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df6aa5c-73f6-4f58-b2b4-aaa19846a614" path="/var/lib/kubelet/pods/3df6aa5c-73f6-4f58-b2b4-aaa19846a614/volumes"
Jul 28 16:47:46.389249 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:46.389207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:46.391882 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:46.391848    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-jmhzp\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:46.572723 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:46.572676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:46.717589 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:46.717565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:47:46.719706 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:47:46.719665    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1903e8ba_4244_46a7_9b76_bf826fea6f57.slice/crio-364604374e117cad49e403f3079432c318b07ed0099ac41e760872f6e6c7f904 WatchSource:0}: Error finding container 364604374e117cad49e403f3079432c318b07ed0099ac41e760872f6e6c7f904: Status 404 returned error can't find the container with id 364604374e117cad49e403f3079432c318b07ed0099ac41e760872f6e6c7f904
Jul 28 16:47:47.723270 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:47.723226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerStarted","Data":"269241797b0439e4b7193e1f7e7f56bb6079c9005f5442da44841caa7c02611b"}
Jul 28 16:47:47.723763 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:47.723294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerStarted","Data":"364604374e117cad49e403f3079432c318b07ed0099ac41e760872f6e6c7f904"}
Jul 28 16:47:51.743895 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:51.743862    2580 generic.go:356] "Generic (PLEG): container finished" podID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerID="269241797b0439e4b7193e1f7e7f56bb6079c9005f5442da44841caa7c02611b" exitCode=0
Jul 28 16:47:51.744310 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:51.743934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerDied","Data":"269241797b0439e4b7193e1f7e7f56bb6079c9005f5442da44841caa7c02611b"}
Jul 28 16:47:51.745299 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:51.745283    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:47:56.773066 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:56.773028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerStarted","Data":"6c932c63c4de275eea1c16c520f8eb6062cb3714bd587dcc569a6032176fc1ea"}
Jul 28 16:47:56.773495 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:56.773089    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerStarted","Data":"5a71fc98e9f8277b5e8f39ceeee39036161826818f185c3bed3776c3223cdd6d"}
Jul 28 16:47:56.773495 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:56.773302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:56.794900 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:56.794848    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podStartSLOduration=2.994714041 podStartE2EDuration="11.794832843s" totalImagesPullingTime="3.961531714s" totalInitContainerRuntime="4.838587088s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:47:56.792292844 +0000 UTC m=+4491.160945773" watchObservedRunningTime="2026-07-28 16:47:56.794832843 +0000 UTC m=+4491.163485759"
Jul 28 16:47:57.777712 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:57.777673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:47:57.778869 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:57.778842    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 16:47:58.782973 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:47:58.782930    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 16:48:03.787558 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:03.787529    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:48:03.788079 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:03.787988    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 16:48:13.788552 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:13.788512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:48:26.825422 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:26.825385    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:48:26.825820 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:26.825719    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" containerID="cri-o://5a71fc98e9f8277b5e8f39ceeee39036161826818f185c3bed3776c3223cdd6d" gracePeriod=30
Jul 28 16:48:26.825820 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:26.825719    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container" containerID="cri-o://6c932c63c4de275eea1c16c520f8eb6062cb3714bd587dcc569a6032176fc1ea" gracePeriod=30
Jul 28 16:48:26.843792 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:26.843753    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:48:27.912416 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:27.912382    2580 generic.go:356] "Generic (PLEG): container finished" podID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerID="5a71fc98e9f8277b5e8f39ceeee39036161826818f185c3bed3776c3223cdd6d" exitCode=2
Jul 28 16:48:27.912802 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:27.912454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerDied","Data":"5a71fc98e9f8277b5e8f39ceeee39036161826818f185c3bed3776c3223cdd6d"}
Jul 28 16:48:28.784149 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:28.784084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:31.979482 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:31.979443    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:48:33.174592 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:33.174554    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:48:33.783766 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:33.783720    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:38.783835 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:38.783792    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:38.784271 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:38.783930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:48:43.783792 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:43.783744    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:48.783591 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:48.783547    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:53.783768 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:53.783725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 16:48:57.046459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.046425    2580 generic.go:356] "Generic (PLEG): container finished" podID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerID="6c932c63c4de275eea1c16c520f8eb6062cb3714bd587dcc569a6032176fc1ea" exitCode=137
Jul 28 16:48:57.046459 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.046468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerDied","Data":"6c932c63c4de275eea1c16c520f8eb6062cb3714bd587dcc569a6032176fc1ea"}
Jul 28 16:48:57.472936 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.472910    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:48:57.620697 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.620604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") pod \"1903e8ba-4244-46a7-9b76-bf826fea6f57\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") "
Jul 28 16:48:57.620697 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.620676    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config\") pod \"1903e8ba-4244-46a7-9b76-bf826fea6f57\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") "
Jul 28 16:48:57.620934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.620713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwhhk\" (UniqueName: \"kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk\") pod \"1903e8ba-4244-46a7-9b76-bf826fea6f57\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") "
Jul 28 16:48:57.620934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.620732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location\") pod \"1903e8ba-4244-46a7-9b76-bf826fea6f57\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") "
Jul 28 16:48:57.620934 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.620751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert\") pod \"1903e8ba-4244-46a7-9b76-bf826fea6f57\" (UID: \"1903e8ba-4244-46a7-9b76-bf826fea6f57\") "
Jul 28 16:48:57.621175 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.621150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1903e8ba-4244-46a7-9b76-bf826fea6f57" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:48:57.621228 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.621189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1903e8ba-4244-46a7-9b76-bf826fea6f57" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:48:57.622992 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.622969    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1903e8ba-4244-46a7-9b76-bf826fea6f57" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:48:57.623090 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.623068    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk" (OuterVolumeSpecName: "kube-api-access-dwhhk") pod "1903e8ba-4244-46a7-9b76-bf826fea6f57" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57"). InnerVolumeSpecName "kube-api-access-dwhhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:48:57.631884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.631856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1903e8ba-4244-46a7-9b76-bf826fea6f57" (UID: "1903e8ba-4244-46a7-9b76-bf826fea6f57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:48:57.721418 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.721383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:48:57.721418 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.721412    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhhk\" (UniqueName: \"kubernetes.io/projected/1903e8ba-4244-46a7-9b76-bf826fea6f57-kube-api-access-dwhhk\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:48:57.721418 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.721424    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903e8ba-4244-46a7-9b76-bf826fea6f57-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:48:57.721657 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.721437    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1903e8ba-4244-46a7-9b76-bf826fea6f57-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:48:57.721657 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:57.721450    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1903e8ba-4244-46a7-9b76-bf826fea6f57-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:48:58.053144 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.053030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp" event={"ID":"1903e8ba-4244-46a7-9b76-bf826fea6f57","Type":"ContainerDied","Data":"364604374e117cad49e403f3079432c318b07ed0099ac41e760872f6e6c7f904"}
Jul 28 16:48:58.053144 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.053109    2580 scope.go:117] "RemoveContainer" containerID="5a71fc98e9f8277b5e8f39ceeee39036161826818f185c3bed3776c3223cdd6d"
Jul 28 16:48:58.053666 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.053055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"
Jul 28 16:48:58.062935 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.062919    2580 scope.go:117] "RemoveContainer" containerID="6c932c63c4de275eea1c16c520f8eb6062cb3714bd587dcc569a6032176fc1ea"
Jul 28 16:48:58.070957 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.070940    2580 scope.go:117] "RemoveContainer" containerID="269241797b0439e4b7193e1f7e7f56bb6079c9005f5442da44841caa7c02611b"
Jul 28 16:48:58.078570 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.078537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:48:58.081960 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.081938    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-jmhzp"]
Jul 28 16:48:58.186001 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:48:58.185972    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" path="/var/lib/kubelet/pods/1903e8ba-4244-46a7-9b76-bf826fea6f57/volumes"
Jul 28 16:49:13.389911 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:49:13.389868    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:49:18.500117 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:49:18.500081    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:49:18.583489 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:49:18.583462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:51:40.059465 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:51:40.059433    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:51:45.105701 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:51:45.105658    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:51:45.182954 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:51:45.182913    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:53:00.612601 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:00.612556    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:53:25.323446 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:25.323410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:53:30.480427 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.480397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:53:30.540146 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540090    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:53:30.540447 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540435    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container"
Jul 28 16:53:30.540510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540448    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container"
Jul 28 16:53:30.540510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy"
Jul 28 16:53:30.540510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy"
Jul 28 16:53:30.540510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540486    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="storage-initializer"
Jul 28 16:53:30.540510 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540492    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="storage-initializer"
Jul 28 16:53:30.540668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kube-rbac-proxy"
Jul 28 16:53:30.540668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.540550    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1903e8ba-4244-46a7-9b76-bf826fea6f57" containerName="kserve-container"
Jul 28 16:53:30.543792 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.543776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.546281 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.546258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 16:53:30.546407 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.546360    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 16:53:30.546666 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.546650    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:53:30.546757 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.546669    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:53:30.547922 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.547891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:53:30.547922 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.547890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:53:30.548106 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.547934    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:53:30.556679 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.556656    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:53:30.623639 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.623611    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:53:30.675722 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.675675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.675884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.675728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.675884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.675756    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.675884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.675816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8tx\" (UniqueName: \"kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.675993 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.675894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.776816 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.776732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8tx\" (UniqueName: \"kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.776816 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.776794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.776994 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.776824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.776994 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.776846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.777188 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.777161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.777494 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.777470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.780168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.778002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.780168 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.778861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.784150 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.780401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.787718 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.787695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8tx\" (UniqueName: \"kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.854409 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.854366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:30.994835 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:30.994810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:53:30.997414 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:53:30.997381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1788a1_9eca_474a_a600_84b74a39e4dc.slice/crio-d8dd5607f09fedffa1ed1850bb813e8cbfa7e8c178986ce51f08a25c7b309bde WatchSource:0}: Error finding container d8dd5607f09fedffa1ed1850bb813e8cbfa7e8c178986ce51f08a25c7b309bde: Status 404 returned error can't find the container with id d8dd5607f09fedffa1ed1850bb813e8cbfa7e8c178986ce51f08a25c7b309bde
Jul 28 16:53:31.219265 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:31.219225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerStarted","Data":"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"}
Jul 28 16:53:31.219437 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:31.219306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerStarted","Data":"d8dd5607f09fedffa1ed1850bb813e8cbfa7e8c178986ce51f08a25c7b309bde"}
Jul 28 16:53:35.236506 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:35.236469    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerID="c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c" exitCode=0
Jul 28 16:53:35.236886 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:35.236541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerDied","Data":"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"}
Jul 28 16:53:36.243051 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:36.243016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerStarted","Data":"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"}
Jul 28 16:53:36.243476 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:36.243081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerStarted","Data":"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"}
Jul 28 16:53:36.243476 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:36.243290    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:36.267221 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:36.267159    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" podStartSLOduration=2.630635355 podStartE2EDuration="6.26712298s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.636487625s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:53:36.266149008 +0000 UTC m=+4830.634801993" watchObservedRunningTime="2026-07-28 16:53:36.26712298 +0000 UTC m=+4830.635775895"
Jul 28 16:53:37.248122 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:37.248089    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:43.258475 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:43.258391    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:53:57.779736 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:53:57.779623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:54:13.262669 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:13.262633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:54:20.542355 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:20.542314    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:54:20.542742 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:20.542553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kserve-container" containerID="cri-o://34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740" gracePeriod=30
Jul 28 16:54:20.542742 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:20.542598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kube-rbac-proxy" containerID="cri-o://01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa" gracePeriod=30
Jul 28 16:54:20.557647 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:20.557620    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:54:21.440877 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:21.440847    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerID="01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa" exitCode=2
Jul 28 16:54:21.441040 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:21.440917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerDied","Data":"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"}
Jul 28 16:54:23.253403 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:23.253362    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Jul 28 16:54:23.259872 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:23.259836    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 28 16:54:25.711752 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.711688    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:54:25.714024 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.714000    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:54:25.717636 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.717620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.720327 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.720305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 16:54:25.720449 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.720307    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 16:54:25.728655 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.728635    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:54:25.806118 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.806072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:54:25.873199 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.873158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.873370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.873209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.873370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.873240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.873370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.873275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jps\" (UniqueName: \"kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.873370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.873332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.974911 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.974820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.974911 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.974864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.974911 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.974884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.975225 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.974917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jps\" (UniqueName: \"kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.975225 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.974938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.975354 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.975329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.975592 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.975572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.975711 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.975690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.977612 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.977593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:25.985361 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:25.985329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jps\" (UniqueName: \"kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:26.028143 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:26.028094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:26.165727 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:26.165703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:54:26.168414 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:54:26.168384    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e76659e_fce0_47db_b7dc_162e81e10850.slice/crio-2a1f0d0e34a09c8e9fe35674e1ad1a54e46019504a7145ad833fd9eb8f976e9f WatchSource:0}: Error finding container 2a1f0d0e34a09c8e9fe35674e1ad1a54e46019504a7145ad833fd9eb8f976e9f: Status 404 returned error can't find the container with id 2a1f0d0e34a09c8e9fe35674e1ad1a54e46019504a7145ad833fd9eb8f976e9f
Jul 28 16:54:26.466104 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:26.466061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerStarted","Data":"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"}
Jul 28 16:54:26.466298 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:26.466160    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerStarted","Data":"2a1f0d0e34a09c8e9fe35674e1ad1a54e46019504a7145ad833fd9eb8f976e9f"}
Jul 28 16:54:27.282876 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.282850    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:54:27.389054 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls\") pod \"6a1788a1-9eca-474a-a600-84b74a39e4dc\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") "
Jul 28 16:54:27.389276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location\") pod \"6a1788a1-9eca-474a-a600-84b74a39e4dc\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") "
Jul 28 16:54:27.389276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config\") pod \"6a1788a1-9eca-474a-a600-84b74a39e4dc\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") "
Jul 28 16:54:27.389276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8tx\" (UniqueName: \"kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx\") pod \"6a1788a1-9eca-474a-a600-84b74a39e4dc\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") "
Jul 28 16:54:27.389276 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert\") pod \"6a1788a1-9eca-474a-a600-84b74a39e4dc\" (UID: \"6a1788a1-9eca-474a-a600-84b74a39e4dc\") "
Jul 28 16:54:27.389509 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a1788a1-9eca-474a-a600-84b74a39e4dc" (UID: "6a1788a1-9eca-474a-a600-84b74a39e4dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:54:27.389582 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a1788a1-9eca-474a-a600-84b74a39e4dc" (UID: "6a1788a1-9eca-474a-a600-84b74a39e4dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:27.389628 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.389612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a1788a1-9eca-474a-a600-84b74a39e4dc" (UID: "6a1788a1-9eca-474a-a600-84b74a39e4dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:54:27.391416 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.391392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a1788a1-9eca-474a-a600-84b74a39e4dc" (UID: "6a1788a1-9eca-474a-a600-84b74a39e4dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:54:27.391472 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.391432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx" (OuterVolumeSpecName: "kube-api-access-nh8tx") pod "6a1788a1-9eca-474a-a600-84b74a39e4dc" (UID: "6a1788a1-9eca-474a-a600-84b74a39e4dc"). InnerVolumeSpecName "kube-api-access-nh8tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:54:27.472145 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.472090    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerID="34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740" exitCode=0
Jul 28 16:54:27.472311 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.472153    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"
Jul 28 16:54:27.472311 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.472172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerDied","Data":"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"}
Jul 28 16:54:27.472311 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.472238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs" event={"ID":"6a1788a1-9eca-474a-a600-84b74a39e4dc","Type":"ContainerDied","Data":"d8dd5607f09fedffa1ed1850bb813e8cbfa7e8c178986ce51f08a25c7b309bde"}
Jul 28 16:54:27.472311 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.472273    2580 scope.go:117] "RemoveContainer" containerID="01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"
Jul 28 16:54:27.481615 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.481595    2580 scope.go:117] "RemoveContainer" containerID="34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"
Jul 28 16:54:27.492641 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.492608    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a1788a1-9eca-474a-a600-84b74a39e4dc-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:54:27.492641 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.492643    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a1788a1-9eca-474a-a600-84b74a39e4dc-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:54:27.492851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.492658    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:54:27.492851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.492673    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8tx\" (UniqueName: \"kubernetes.io/projected/6a1788a1-9eca-474a-a600-84b74a39e4dc-kube-api-access-nh8tx\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:54:27.492851 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.492687    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a1788a1-9eca-474a-a600-84b74a39e4dc-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:54:27.496328 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.496303    2580 scope.go:117] "RemoveContainer" containerID="c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"
Jul 28 16:54:27.498167 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.498120    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:54:27.502391 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.502371    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fcdfs"]
Jul 28 16:54:27.506339 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.506323    2580 scope.go:117] "RemoveContainer" containerID="01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"
Jul 28 16:54:27.506615 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:54:27.506589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa\": container with ID starting with 01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa not found: ID does not exist" containerID="01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"
Jul 28 16:54:27.506670 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.506628    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa"} err="failed to get container status \"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa\": rpc error: code = NotFound desc = could not find container \"01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa\": container with ID starting with 01f3cd3621fe9406cf944b66f09502662a54e38693b6b1e0a8df071245e95caa not found: ID does not exist"
Jul 28 16:54:27.506670 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.506654    2580 scope.go:117] "RemoveContainer" containerID="34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"
Jul 28 16:54:27.506863 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:54:27.506845    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740\": container with ID starting with 34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740 not found: ID does not exist" containerID="34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"
Jul 28 16:54:27.506918 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.506870    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740"} err="failed to get container status \"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740\": rpc error: code = NotFound desc = could not find container \"34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740\": container with ID starting with 34e263a9c1f0ab3a518c53e821d9798460dc90f3c8e097119a0fa96910b4e740 not found: ID does not exist"
Jul 28 16:54:27.506918 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.506889    2580 scope.go:117] "RemoveContainer" containerID="c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"
Jul 28 16:54:27.507139 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:54:27.507110    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c\": container with ID starting with c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c not found: ID does not exist" containerID="c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"
Jul 28 16:54:27.507201 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:27.507179    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c"} err="failed to get container status \"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c\": rpc error: code = NotFound desc = could not find container \"c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c\": container with ID starting with c1bb87d16b3cf963237a075bc137a25699ff8431acca850fe71e9baa9a16239c not found: ID does not exist"
Jul 28 16:54:28.185909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:28.185872    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" path="/var/lib/kubelet/pods/6a1788a1-9eca-474a-a600-84b74a39e4dc/volumes"
Jul 28 16:54:30.489151 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:30.489083    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e76659e-fce0-47db-b7dc-162e81e10850" containerID="ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c" exitCode=0
Jul 28 16:54:30.489588 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:30.489158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerDied","Data":"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"}
Jul 28 16:54:31.495988 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:31.495956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerStarted","Data":"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"}
Jul 28 16:54:31.496396 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:31.496017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerStarted","Data":"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"}
Jul 28 16:54:31.496396 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:31.496222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:31.496396 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:31.496276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:54:31.520844 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:31.520790    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" podStartSLOduration=2.683147274 podStartE2EDuration="6.520774921s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837627647s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:54:31.520074936 +0000 UTC m=+4885.888727853" watchObservedRunningTime="2026-07-28 16:54:31.520774921 +0000 UTC m=+4885.889427837"
Jul 28 16:54:37.513007 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:54:37.512970    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:55:07.517641 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:07.517614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:55:15.823052 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:15.823012    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:55:15.823588 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:15.823280    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kserve-container" containerID="cri-o://078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba" gracePeriod=30
Jul 28 16:55:15.823588 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:15.823298    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kube-rbac-proxy" containerID="cri-o://bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52" gracePeriod=30
Jul 28 16:55:15.855334 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:15.855303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:55:16.707728 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:16.707694    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e76659e-fce0-47db-b7dc-162e81e10850" containerID="bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52" exitCode=2
Jul 28 16:55:16.707909 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:16.707756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerDied","Data":"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"}
Jul 28 16:55:17.507758 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:17.507702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Jul 28 16:55:20.965194 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:20.965153    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:55:21.262441 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:21.262352    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:55:22.375012 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.374986    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:55:22.465849 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.465762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls\") pod \"7e76659e-fce0-47db-b7dc-162e81e10850\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") "
Jul 28 16:55:22.465849 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.465834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config\") pod \"7e76659e-fce0-47db-b7dc-162e81e10850\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") "
Jul 28 16:55:22.466089 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.465868    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location\") pod \"7e76659e-fce0-47db-b7dc-162e81e10850\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") "
Jul 28 16:55:22.466089 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.465894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert\") pod \"7e76659e-fce0-47db-b7dc-162e81e10850\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") "
Jul 28 16:55:22.466089 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.465921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4jps\" (UniqueName: \"kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps\") pod \"7e76659e-fce0-47db-b7dc-162e81e10850\" (UID: \"7e76659e-fce0-47db-b7dc-162e81e10850\") "
Jul 28 16:55:22.466281 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.466230    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e76659e-fce0-47db-b7dc-162e81e10850" (UID: "7e76659e-fce0-47db-b7dc-162e81e10850"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:55:22.466281 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.466247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e76659e-fce0-47db-b7dc-162e81e10850" (UID: "7e76659e-fce0-47db-b7dc-162e81e10850"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:55:22.466376 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.466279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e76659e-fce0-47db-b7dc-162e81e10850" (UID: "7e76659e-fce0-47db-b7dc-162e81e10850"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:55:22.467969 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.467951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e76659e-fce0-47db-b7dc-162e81e10850" (UID: "7e76659e-fce0-47db-b7dc-162e81e10850"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:55:22.468103 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.468083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps" (OuterVolumeSpecName: "kube-api-access-h4jps") pod "7e76659e-fce0-47db-b7dc-162e81e10850" (UID: "7e76659e-fce0-47db-b7dc-162e81e10850"). InnerVolumeSpecName "kube-api-access-h4jps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:55:22.566646 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.566607    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:55:22.566646 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.566636    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e76659e-fce0-47db-b7dc-162e81e10850-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:55:22.566646 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.566646    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e76659e-fce0-47db-b7dc-162e81e10850-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:55:22.566646 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.566657    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jps\" (UniqueName: \"kubernetes.io/projected/7e76659e-fce0-47db-b7dc-162e81e10850-kube-api-access-h4jps\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:55:22.566907 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.566667    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e76659e-fce0-47db-b7dc-162e81e10850-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:55:22.736309 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.736210    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e76659e-fce0-47db-b7dc-162e81e10850" containerID="078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba" exitCode=0
Jul 28 16:55:22.736309 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.736257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerDied","Data":"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"}
Jul 28 16:55:22.736542 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.736314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2" event={"ID":"7e76659e-fce0-47db-b7dc-162e81e10850","Type":"ContainerDied","Data":"2a1f0d0e34a09c8e9fe35674e1ad1a54e46019504a7145ad833fd9eb8f976e9f"}
Jul 28 16:55:22.736542 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.736264    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"
Jul 28 16:55:22.736542 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.736353    2580 scope.go:117] "RemoveContainer" containerID="bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"
Jul 28 16:55:22.745748 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.745728    2580 scope.go:117] "RemoveContainer" containerID="078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"
Jul 28 16:55:22.759277 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.759251    2580 scope.go:117] "RemoveContainer" containerID="ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"
Jul 28 16:55:22.760579 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.760551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:55:22.764788 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.764766    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-fjvp2"]
Jul 28 16:55:22.767638 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.767622    2580 scope.go:117] "RemoveContainer" containerID="bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"
Jul 28 16:55:22.767954 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:55:22.767933    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52\": container with ID starting with bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52 not found: ID does not exist" containerID="bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"
Jul 28 16:55:22.768004 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.767963    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52"} err="failed to get container status \"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52\": rpc error: code = NotFound desc = could not find container \"bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52\": container with ID starting with bfaeb7a3346cb05ce88910c9734ce9e12fceefd2e4bdd211e9b7f8935f8d6d52 not found: ID does not exist"
Jul 28 16:55:22.768004 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.767981    2580 scope.go:117] "RemoveContainer" containerID="078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"
Jul 28 16:55:22.768280 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:55:22.768262    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba\": container with ID starting with 078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba not found: ID does not exist" containerID="078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"
Jul 28 16:55:22.768370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.768288    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba"} err="failed to get container status \"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba\": rpc error: code = NotFound desc = could not find container \"078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba\": container with ID starting with 078e17384a9b74399cb2f0ab77aeb0d187a09786235f42716c28d0bbebcd32ba not found: ID does not exist"
Jul 28 16:55:22.768370 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.768310    2580 scope.go:117] "RemoveContainer" containerID="ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"
Jul 28 16:55:22.768573 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:55:22.768556    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c\": container with ID starting with ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c not found: ID does not exist" containerID="ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"
Jul 28 16:55:22.768611 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:22.768579    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c"} err="failed to get container status \"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c\": rpc error: code = NotFound desc = could not find container \"ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c\": container with ID starting with ddb036a0c7d67b66b3c3464b07f7b4f141f1b4c2e478264fefae966c87ef0b7c not found: ID does not exist"
Jul 28 16:55:24.185857 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:55:24.185820    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" path="/var/lib/kubelet/pods/7e76659e-fce0-47db-b7dc-162e81e10850/volumes"
Jul 28 16:56:41.123987 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:41.123933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:56:46.245672 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.245640    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:56:46.280111 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280075    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:56:46.280471 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280455    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280473    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280482    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280487    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280496    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kserve-container"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280502    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kserve-container"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kserve-container"
Jul 28 16:56:46.280524 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280515    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kserve-container"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280528    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="storage-initializer"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280534    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="storage-initializer"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280544    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="storage-initializer"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="storage-initializer"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280607    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kserve-container"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280614    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e76659e-fce0-47db-b7dc-162e81e10850" containerName="kube-rbac-proxy"
Jul 28 16:56:46.280771 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.280623    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1788a1-9eca-474a-a600-84b74a39e4dc" containerName="kserve-container"
Jul 28 16:56:46.283775 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.283757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.286449 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286403    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:56:46.286586 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:56:46.286674 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 16:56:46.286752 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:56:46.286802 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 16:56:46.286880 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.286863    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:56:46.287841 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.287825    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mth8l\""
Jul 28 16:56:46.293726 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.293702    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:56:46.377598 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.377561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.377753 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.377616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.377753 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.377684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.377830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.377758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxgk6\" (UniqueName: \"kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.377830 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.377793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.478691 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.478656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxgk6\" (UniqueName: \"kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.478879 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.478713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.478879 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.478746    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.478879 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.478773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.478879 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.478800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.479365 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.479342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.479576 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.479556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.479576 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.479568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.481606 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.481582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.487456 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.487432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxgk6\" (UniqueName: \"kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.555341 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.555253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:56:46.594984 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.594950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:46.733201 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:46.733170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:56:46.735854 ip-10-0-140-97 kubenswrapper[2580]: W0728 16:56:46.735809    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3acdba_1cb1_464d_a3c8_eafd3d11fd65.slice/crio-38b088d878c878bd6de64a9484946b1185aef93a3b0e62bd339dce08b9d6f3d0 WatchSource:0}: Error finding container 38b088d878c878bd6de64a9484946b1185aef93a3b0e62bd339dce08b9d6f3d0: Status 404 returned error can't find the container with id 38b088d878c878bd6de64a9484946b1185aef93a3b0e62bd339dce08b9d6f3d0
Jul 28 16:56:47.093653 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:47.093612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerStarted","Data":"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"}
Jul 28 16:56:47.093833 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:47.093678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerStarted","Data":"38b088d878c878bd6de64a9484946b1185aef93a3b0e62bd339dce08b9d6f3d0"}
Jul 28 16:56:51.112411 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:51.112373    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerID="5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe" exitCode=0
Jul 28 16:56:51.112785 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:51.112447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerDied","Data":"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"}
Jul 28 16:56:52.118970 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:52.118934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerStarted","Data":"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"}
Jul 28 16:56:52.119419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:52.119003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerStarted","Data":"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"}
Jul 28 16:56:52.119419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:52.119274    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:52.142557 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:52.142505    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" podStartSLOduration=2.374714962 podStartE2EDuration="6.142489767s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767774805s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:56:52.140700629 +0000 UTC m=+5026.509353547" watchObservedRunningTime="2026-07-28 16:56:52.142489767 +0000 UTC m=+5026.511142683"
Jul 28 16:56:53.123889 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:53.123853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:56:59.133257 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:56:59.133169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:57:29.165023 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:29.164972    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 16:57:39.136295 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:39.136251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:57:46.375229 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:46.375196    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:57:46.375743 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:46.375442    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kserve-container" containerID="cri-o://401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635" gracePeriod=30
Jul 28 16:57:46.375743 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:46.375494    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kube-rbac-proxy" containerID="cri-o://44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767" gracePeriod=30
Jul 28 16:57:46.408552 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:46.408520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:57:47.364364 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:47.364331    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerID="44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767" exitCode=2
Jul 28 16:57:47.364580 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:47.364409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerDied","Data":"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"}
Jul 28 16:57:49.128197 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:49.128116    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Jul 28 16:57:52.309567 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:52.309531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:57:52.410833 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:52.410789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:57:54.026969 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.026936    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:57:54.180797 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.180693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert\") pod \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") "
Jul 28 16:57:54.180797 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.180734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxgk6\" (UniqueName: \"kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6\") pod \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") "
Jul 28 16:57:54.180797 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.180801    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config\") pod \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") "
Jul 28 16:57:54.181108 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.180830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location\") pod \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") "
Jul 28 16:57:54.181108 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.180896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls\") pod \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\" (UID: \"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65\") "
Jul 28 16:57:54.181265 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.181180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" (UID: "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:57:54.181265 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.181235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" (UID: "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:57:54.181265 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.181246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" (UID: "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:57:54.183207 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.183171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" (UID: "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:57:54.183345 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.183247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6" (OuterVolumeSpecName: "kube-api-access-cxgk6") pod "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" (UID: "5f3acdba-1cb1-464d-a3c8-eafd3d11fd65"). InnerVolumeSpecName "kube-api-access-cxgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:57:54.282419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.282387    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:57:54.282419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.282416    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kserve-provision-location\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:57:54.282419 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.282426    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-proxy-tls\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:57:54.282668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.282435    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-cabundle-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:57:54.282668 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.282444    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxgk6\" (UniqueName: \"kubernetes.io/projected/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65-kube-api-access-cxgk6\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 16:57:54.402482 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.402443    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerID="401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635" exitCode=0
Jul 28 16:57:54.402675 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.402500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerDied","Data":"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"}
Jul 28 16:57:54.402675 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.402546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh" event={"ID":"5f3acdba-1cb1-464d-a3c8-eafd3d11fd65","Type":"ContainerDied","Data":"38b088d878c878bd6de64a9484946b1185aef93a3b0e62bd339dce08b9d6f3d0"}
Jul 28 16:57:54.402675 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.402572    2580 scope.go:117] "RemoveContainer" containerID="44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"
Jul 28 16:57:54.402675 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.402501    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"
Jul 28 16:57:54.411296 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.411276    2580 scope.go:117] "RemoveContainer" containerID="401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"
Jul 28 16:57:54.419169 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.419151    2580 scope.go:117] "RemoveContainer" containerID="5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"
Jul 28 16:57:54.426845 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.426828    2580 scope.go:117] "RemoveContainer" containerID="44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"
Jul 28 16:57:54.427102 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:57:54.427079    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767\": container with ID starting with 44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767 not found: ID does not exist" containerID="44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"
Jul 28 16:57:54.427261 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.427107    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767"} err="failed to get container status \"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767\": rpc error: code = NotFound desc = could not find container \"44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767\": container with ID starting with 44aa511c5958dd71f4f3d8a87a5902bebb293537a5eddb66c676cb0cf0c3e767 not found: ID does not exist"
Jul 28 16:57:54.427261 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.427141    2580 scope.go:117] "RemoveContainer" containerID="401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"
Jul 28 16:57:54.427381 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:57:54.427363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635\": container with ID starting with 401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635 not found: ID does not exist" containerID="401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"
Jul 28 16:57:54.427420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.427388    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635"} err="failed to get container status \"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635\": rpc error: code = NotFound desc = could not find container \"401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635\": container with ID starting with 401a983c74a485e955bd7446a0169beeefaf54d110743252d1d13fa3bd3fb635 not found: ID does not exist"
Jul 28 16:57:54.427420 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.427404    2580 scope.go:117] "RemoveContainer" containerID="5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"
Jul 28 16:57:54.427600 ip-10-0-140-97 kubenswrapper[2580]: E0728 16:57:54.427584    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe\": container with ID starting with 5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe not found: ID does not exist" containerID="5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"
Jul 28 16:57:54.427639 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.427606    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe"} err="failed to get container status \"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe\": rpc error: code = NotFound desc = could not find container \"5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe\": container with ID starting with 5936b8f5bf1e5facd02bd7cb505030fabce1c5b659685339b5664dfbe03f62fe not found: ID does not exist"
Jul 28 16:57:54.433253 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.433202    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:57:54.437939 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:54.437919    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-msbxh"]
Jul 28 16:57:56.186365 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:57:56.186321    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" path="/var/lib/kubelet/pods/5f3acdba-1cb1-464d-a3c8-eafd3d11fd65/volumes"
Jul 28 16:59:11.695672 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:59:11.695627    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:59:16.829446 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:59:16.829411    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 16:59:17.351884 ip-10-0-140-97 kubenswrapper[2580]: I0728 16:59:17.351844    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:00:37.076181 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:00:37.076118    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:00:37.156147 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:00:37.156100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:00:37.744483 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:00:37.744447    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:01:03.787445 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:01:03.787404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:01:57.928364 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:01:57.928324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:07.229199 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:07.229165    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:08.287196 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:08.287161    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:08.643529 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:08.643492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:18.357838 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:18.357801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:19.467174 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:19.467116    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:02:19.602481 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:02:19.602442    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:03:49.628567 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:03:49.628531    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:03:50.607602 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:03:50.607561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:03:50.974584 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:03:50.974490    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:04:00.666560 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:04:00.666520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:04:01.769345 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:04:01.769313    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:04:01.867481 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:04:01.867439    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:11.324874 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:11.324831    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:31.914585 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:31.914536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:32.888098 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:32.888064    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:33.596269 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:33.596236    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:42.945103 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:42.945065    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:44.774574 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774544    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z5g7h/must-gather-9v9jn"]
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774878    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kube-rbac-proxy"
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774888    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kube-rbac-proxy"
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="storage-initializer"
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774915    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="storage-initializer"
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774923    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kserve-container"
Jul 28 17:05:44.774968 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774928    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kserve-container"
Jul 28 17:05:44.775195 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774982    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kube-rbac-proxy"
Jul 28 17:05:44.775195 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.774992    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3acdba-1cb1-464d-a3c8-eafd3d11fd65" containerName="kserve-container"
Jul 28 17:05:44.778428 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.778406    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:44.781195 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.781173    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5g7h\"/\"kube-root-ca.crt\""
Jul 28 17:05:44.781326 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.781215    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5g7h\"/\"openshift-service-ca.crt\""
Jul 28 17:05:44.786379 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.786354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5g7h/must-gather-9v9jn"]
Jul 28 17:05:44.806072 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.806039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:44.888650 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.888617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:44.888820 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.888693    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgksr\" (UniqueName: \"kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:44.903173 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.903145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:05:44.989971 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.989929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:44.990200 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.990025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgksr\" (UniqueName: \"kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:44.990309 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:44.990290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:45.001788 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:45.001764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgksr\" (UniqueName: \"kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr\") pod \"must-gather-9v9jn\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") " pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:45.099041 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:45.099009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:05:45.232763 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:45.232717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5g7h/must-gather-9v9jn"]
Jul 28 17:05:45.235086 ip-10-0-140-97 kubenswrapper[2580]: W0728 17:05:45.235055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb36f44c8_a614_4715_ab18_00920baed5f6.slice/crio-ef78c037ff6780b5874535212a56b64b5fd7467615268895cdf08560fbe306c3 WatchSource:0}: Error finding container ef78c037ff6780b5874535212a56b64b5fd7467615268895cdf08560fbe306c3: Status 404 returned error can't find the container with id ef78c037ff6780b5874535212a56b64b5fd7467615268895cdf08560fbe306c3
Jul 28 17:05:45.236847 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:45.236820    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:05:45.375625 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:45.375533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" event={"ID":"b36f44c8-a614-4715-ab18-00920baed5f6","Type":"ContainerStarted","Data":"ef78c037ff6780b5874535212a56b64b5fd7467615268895cdf08560fbe306c3"}
Jul 28 17:05:52.413972 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:52.413933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" event={"ID":"b36f44c8-a614-4715-ab18-00920baed5f6","Type":"ContainerStarted","Data":"92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae"}
Jul 28 17:05:52.413972 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:52.413980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" event={"ID":"b36f44c8-a614-4715-ab18-00920baed5f6","Type":"ContainerStarted","Data":"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"}
Jul 28 17:05:52.434081 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:05:52.434021    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" podStartSLOduration=1.589295181 podStartE2EDuration="8.433999702s" totalImagesPullingTime="6.844704521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:05:52.432012071 +0000 UTC m=+5566.800664988" watchObservedRunningTime="2026-07-28 17:05:52.433999702 +0000 UTC m=+5566.802652618"
Jul 28 17:06:08.648541 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:08.648455    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:19.541112 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:19.541073    2580 generic.go:356] "Generic (PLEG): container finished" podID="b36f44c8-a614-4715-ab18-00920baed5f6" containerID="bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e" exitCode=0
Jul 28 17:06:19.541557 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:19.541168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" event={"ID":"b36f44c8-a614-4715-ab18-00920baed5f6","Type":"ContainerDied","Data":"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"}
Jul 28 17:06:19.541557 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:19.541527    2580 scope.go:117] "RemoveContainer" containerID="bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"
Jul 28 17:06:19.967903 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:19.967866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:19.984043 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:19.984011    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z5g7h_must-gather-9v9jn_b36f44c8-a614-4715-ab18-00920baed5f6/gather/0.log"
Jul 28 17:06:20.566648 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:20.566616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:20.736523 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:20.736486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:20.798353 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:20.798315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:23.504547 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:23.504514    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jstzl_3734d978-9bc8-4025-99ee-f847754332e5/global-pull-secret-syncer/0.log"
Jul 28 17:06:23.706745 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:23.706712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log"
Jul 28 17:06:23.918729 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:23.918701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-97.ec2.internal_969cac25ca034fa30dfc485a33f781d6/haproxy/0.log"
Jul 28 17:06:25.546957 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.546921    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-z5g7h/must-gather-9v9jn"]
Jul 28 17:06:25.547442 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.547116    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-z5g7h/must-gather-9v9jn" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="copy" containerID="cri-o://92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae" gracePeriod=2
Jul 28 17:06:25.550053 ip-10-0-140-97 kubenswrapper[2580]: E0728 17:06:25.550018    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9v9jn\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z5g7h\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:06:25.551582 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.551223    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-z5g7h/must-gather-9v9jn"]
Jul 28 17:06:25.781349 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.781327    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z5g7h_must-gather-9v9jn_b36f44c8-a614-4715-ab18-00920baed5f6/copy/0.log"
Jul 28 17:06:25.783147 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.783117    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:06:25.785437 ip-10-0-140-97 kubenswrapper[2580]: E0728 17:06:25.785412    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9v9jn\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z5g7h\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:06:25.866624 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.866590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output\") pod \"b36f44c8-a614-4715-ab18-00920baed5f6\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") "
Jul 28 17:06:25.866624 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.866628    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgksr\" (UniqueName: \"kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr\") pod \"b36f44c8-a614-4715-ab18-00920baed5f6\" (UID: \"b36f44c8-a614-4715-ab18-00920baed5f6\") "
Jul 28 17:06:25.868312 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.868286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b36f44c8-a614-4715-ab18-00920baed5f6" (UID: "b36f44c8-a614-4715-ab18-00920baed5f6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:06:25.868972 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.868944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr" (OuterVolumeSpecName: "kube-api-access-qgksr") pod "b36f44c8-a614-4715-ab18-00920baed5f6" (UID: "b36f44c8-a614-4715-ab18-00920baed5f6"). InnerVolumeSpecName "kube-api-access-qgksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:06:25.967901 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.967846    2580 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b36f44c8-a614-4715-ab18-00920baed5f6-must-gather-output\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 17:06:25.967901 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:25.967895    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgksr\" (UniqueName: \"kubernetes.io/projected/b36f44c8-a614-4715-ab18-00920baed5f6-kube-api-access-qgksr\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\""
Jul 28 17:06:26.186152 ip-10-0-140-97 kubenswrapper[2580]: E0728 17:06:26.186035    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9v9jn\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z5g7h\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:06:26.186437 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.186418    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" path="/var/lib/kubelet/pods/b36f44c8-a614-4715-ab18-00920baed5f6/volumes"
Jul 28 17:06:26.453923 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.453837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:26.554287 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.554240    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbx6n_ff52c32e-08f8-4417-bda9-9eddd9a13efc/konnectivity-agent/0.log" limit=1024
Jul 28 17:06:26.578819 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.578793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z5g7h_must-gather-9v9jn_b36f44c8-a614-4715-ab18-00920baed5f6/copy/0.log"
Jul 28 17:06:26.579164 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.579116    2580 generic.go:356] "Generic (PLEG): container finished" podID="b36f44c8-a614-4715-ab18-00920baed5f6" containerID="92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae" exitCode=143
Jul 28 17:06:26.579252 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.579181    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5g7h/must-gather-9v9jn"
Jul 28 17:06:26.579302 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.579247    2580 scope.go:117] "RemoveContainer" containerID="92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae"
Jul 28 17:06:26.587400 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.587381    2580 scope.go:117] "RemoveContainer" containerID="bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"
Jul 28 17:06:26.600876 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.600858    2580 scope.go:117] "RemoveContainer" containerID="92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae"
Jul 28 17:06:26.601141 ip-10-0-140-97 kubenswrapper[2580]: E0728 17:06:26.601107    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae\": container with ID starting with 92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae not found: ID does not exist" containerID="92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae"
Jul 28 17:06:26.601247 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.601149    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae"} err="failed to get container status \"92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae\": rpc error: code = NotFound desc = could not find container \"92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae\": container with ID starting with 92b817670901366003dc76cb4447dfcc69e77dbd1a0c9ded94c4a8242945b6ae not found: ID does not exist"
Jul 28 17:06:26.601247 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.601168    2580 scope.go:117] "RemoveContainer" containerID="bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"
Jul 28 17:06:26.601443 ip-10-0-140-97 kubenswrapper[2580]: E0728 17:06:26.601430    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e\": container with ID starting with bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e not found: ID does not exist" containerID="bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"
Jul 28 17:06:26.601496 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:26.601446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e"} err="failed to get container status \"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e\": rpc error: code = NotFound desc = could not find container \"bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e\": container with ID starting with bb724091c5ba051e506cbc29aa30a07076e54c1f5c17c2118eee66160b5c8c8e not found: ID does not exist"
Jul 28 17:06:27.527496 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.527467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/alertmanager/0.log"
Jul 28 17:06:27.565468 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.565433    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/config-reloader/0.log"
Jul 28 17:06:27.612880 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.612857    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/kube-rbac-proxy-web/0.log"
Jul 28 17:06:27.647047 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.647017    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/kube-rbac-proxy/0.log"
Jul 28 17:06:27.688519 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.688488    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/kube-rbac-proxy-metric/0.log"
Jul 28 17:06:27.723290 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.723256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/prom-label-proxy/0.log"
Jul 28 17:06:27.834229 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.834202    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd624046-1670-4924-98a5-58afbe7ade2a/init-config-reloader/0.log"
Jul 28 17:06:27.975892 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:27.975863    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ks29s_025f76d5-e003-4509-b00b-46bf82dfce35/kube-state-metrics/0.log"
Jul 28 17:06:28.000566 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.000541    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ks29s_025f76d5-e003-4509-b00b-46bf82dfce35/kube-rbac-proxy-main/0.log"
Jul 28 17:06:28.032352 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.032324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ks29s_025f76d5-e003-4509-b00b-46bf82dfce35/kube-rbac-proxy-self/0.log"
Jul 28 17:06:28.076379 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.076347    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b97796b5-xrvvh_549cd043-a164-4b0a-be24-a4260992053f/metrics-server/0.log"
Jul 28 17:06:28.122349 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.122275    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-sdvdz_2a8c1491-a611-40f0-a274-a345088606ee/monitoring-plugin/0.log"
Jul 28 17:06:28.260876 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.260845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwt29_27598466-8844-4bff-af12-08a40ef6337f/node-exporter/0.log"
Jul 28 17:06:28.298683 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.298655    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwt29_27598466-8844-4bff-af12-08a40ef6337f/kube-rbac-proxy/0.log"
Jul 28 17:06:28.336054 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.336026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwt29_27598466-8844-4bff-af12-08a40ef6337f/init-textfile/0.log"
Jul 28 17:06:28.481054 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.481019    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pvzsl_4d7ad692-43eb-497f-a82c-29f0f811e276/kube-rbac-proxy-main/0.log"
Jul 28 17:06:28.508182 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.508153    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pvzsl_4d7ad692-43eb-497f-a82c-29f0f811e276/kube-rbac-proxy-self/0.log"
Jul 28 17:06:28.540995 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.540963    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-pvzsl_4d7ad692-43eb-497f-a82c-29f0f811e276/openshift-state-metrics/0.log"
Jul 28 17:06:28.615605 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.615573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/prometheus/0.log"
Jul 28 17:06:28.636348 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.636325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/config-reloader/0.log"
Jul 28 17:06:28.663480 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.663441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/thanos-sidecar/0.log"
Jul 28 17:06:28.693453 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.693426    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/kube-rbac-proxy-web/0.log"
Jul 28 17:06:28.719932 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.719856    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/kube-rbac-proxy/0.log"
Jul 28 17:06:28.749622 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.749579    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/kube-rbac-proxy-thanos/0.log"
Jul 28 17:06:28.778943 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.778913    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1d2838cb-f368-421a-b994-ec4d0172ddd0/init-config-reloader/0.log"
Jul 28 17:06:28.816840 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.816812    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-q9g5g_616bd69c-c54b-4977-a225-afbbff19ec3c/prometheus-operator/0.log"
Jul 28 17:06:28.838066 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.838043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-q9g5g_616bd69c-c54b-4977-a225-afbbff19ec3c/kube-rbac-proxy/0.log"
Jul 28 17:06:28.872426 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.872400    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-t2ghz_820c66c6-0ee9-41f3-9090-a8e676b3a42a/prometheus-operator-admission-webhook/0.log"
Jul 28 17:06:28.906705 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.906673    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-749f86dc87-5mwwf_b8671285-9c62-4350-a9e1-e3e728409d0f/telemeter-client/0.log"
Jul 28 17:06:28.933957 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.933929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-749f86dc87-5mwwf_b8671285-9c62-4350-a9e1-e3e728409d0f/reload/0.log"
Jul 28 17:06:28.962084 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:28.962062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-749f86dc87-5mwwf_b8671285-9c62-4350-a9e1-e3e728409d0f/kube-rbac-proxy/0.log"
Jul 28 17:06:29.000789 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.000698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/thanos-query/0.log"
Jul 28 17:06:29.039911 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.039879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/kube-rbac-proxy-web/0.log"
Jul 28 17:06:29.066877 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.066850    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/kube-rbac-proxy/0.log"
Jul 28 17:06:29.096214 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.096185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/prom-label-proxy/0.log"
Jul 28 17:06:29.122092 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.122056    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/kube-rbac-proxy-rules/0.log"
Jul 28 17:06:29.149353 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:29.149315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dfc4f7648-jfsnc_fbd3f6c6-6a21-4a33-aa90-cfe823272b5f/kube-rbac-proxy-metrics/0.log"
Jul 28 17:06:32.077355 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077272    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"]
Jul 28 17:06:32.077759 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077740    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="gather"
Jul 28 17:06:32.077803 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077761    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="gather"
Jul 28 17:06:32.077803 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077781    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="copy"
Jul 28 17:06:32.077803 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077787    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="copy"
Jul 28 17:06:32.077940 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077847    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="copy"
Jul 28 17:06:32.077940 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.077861    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36f44c8-a614-4715-ab18-00920baed5f6" containerName="gather"
Jul 28 17:06:32.081697 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.081673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.084341 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.084321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qccvn\"/\"default-dockercfg-lxx9f\""
Jul 28 17:06:32.084518 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.084506    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qccvn\"/\"openshift-service-ca.crt\""
Jul 28 17:06:32.085893 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.085878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qccvn\"/\"kube-root-ca.crt\""
Jul 28 17:06:32.103043 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.103017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"]
Jul 28 17:06:32.225428 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.225393    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-sys\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.225618 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.225437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-podres\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.225618 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.225461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779gb\" (UniqueName: \"kubernetes.io/projected/41937ccd-f189-4818-a54f-f406d282326c-kube-api-access-779gb\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.225618 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.225484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-lib-modules\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.225618 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.225533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-proc\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.326961 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.326920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-podres\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.326961 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.326965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779gb\" (UniqueName: \"kubernetes.io/projected/41937ccd-f189-4818-a54f-f406d282326c-kube-api-access-779gb\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.326996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-lib-modules\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-proc\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-podres\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-sys\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-proc\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-lib-modules\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.327284 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.327251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41937ccd-f189-4818-a54f-f406d282326c-sys\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.336949 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.336928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779gb\" (UniqueName: \"kubernetes.io/projected/41937ccd-f189-4818-a54f-f406d282326c-kube-api-access-779gb\") pod \"perf-node-gather-daemonset-xw6hz\" (UID: \"41937ccd-f189-4818-a54f-f406d282326c\") " pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.393110 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.393080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.526438 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.526414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"]
Jul 28 17:06:32.528536 ip-10-0-140-97 kubenswrapper[2580]: W0728 17:06:32.528509    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41937ccd_f189_4818_a54f_f406d282326c.slice/crio-0287b140dee45a58ab85606dc3c9f8c7250b7a30d25064e3c733bd2b1e276da7 WatchSource:0}: Error finding container 0287b140dee45a58ab85606dc3c9f8c7250b7a30d25064e3c733bd2b1e276da7: Status 404 returned error can't find the container with id 0287b140dee45a58ab85606dc3c9f8c7250b7a30d25064e3c733bd2b1e276da7
Jul 28 17:06:32.607344 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.607266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz" event={"ID":"41937ccd-f189-4818-a54f-f406d282326c","Type":"ContainerStarted","Data":"a4707504dce26503b5cb4960e8cba422d32b561e7fda8edcbe86171f93c33f21"}
Jul 28 17:06:32.607344 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.607324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz" event={"ID":"41937ccd-f189-4818-a54f-f406d282326c","Type":"ContainerStarted","Data":"0287b140dee45a58ab85606dc3c9f8c7250b7a30d25064e3c733bd2b1e276da7"}
Jul 28 17:06:32.607344 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.607344    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:32.629380 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:32.629337    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz" podStartSLOduration=0.629319482 podStartE2EDuration="629.319482ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:06:32.627706755 +0000 UTC m=+5606.996359649" watchObservedRunningTime="2026-07-28 17:06:32.629319482 +0000 UTC m=+5606.997972397"
Jul 28 17:06:33.105467 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:33.105435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcj9m_3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b/dns/0.log"
Jul 28 17:06:33.132357 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:33.132329    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fcj9m_3b6178ec-7634-41cf-8bc4-0d5eec6a7d2b/kube-rbac-proxy/0.log"
Jul 28 17:06:33.341634 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:33.341600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z7bcq_119e856f-abfc-42cd-9d2c-052ed52df947/dns-node-resolver/0.log"
Jul 28 17:06:34.007072 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:34.007037    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qn2n4_aeaab55a-2ceb-4543-a282-3a3ebebcb0af/node-ca/0.log"
Jul 28 17:06:35.333345 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:35.333310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fkh4t_12bc6b40-3bc3-4c72-8a9a-2320ec05806e/serve-healthcheck-canary/0.log"
Jul 28 17:06:35.888962 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:35.888932    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jhq_03d46c54-4f83-41a9-9d99-e2f88aec3f4a/kube-rbac-proxy/0.log"
Jul 28 17:06:35.915427 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:35.915394    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jhq_03d46c54-4f83-41a9-9d99-e2f88aec3f4a/exporter/0.log"
Jul 28 17:06:35.942233 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:35.942206    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6jhq_03d46c54-4f83-41a9-9d99-e2f88aec3f4a/extractor/0.log"
Jul 28 17:06:38.621954 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:38.621927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qccvn/perf-node-gather-daemonset-xw6hz"
Jul 28 17:06:38.630211 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:38.630189    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-krzcb_87582518-d6d0-4fdb-8526-954322bbddd2/server/0.log"
Jul 28 17:06:39.052073 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.051992    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9khhl_1ace3bec-23ac-4f63-97f9-492078ce9200/s3-init/0.log"
Jul 28 17:06:39.089235 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.089204    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9khhl_1ace3bec-23ac-4f63-97f9-492078ce9200/download-hf-model/0.log"
Jul 28 17:06:39.133675 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.133645    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-w64h8_cf8fb96f-7eea-4717-a0b7-5544121e1bdc/s3-tls-init-custom/0.log"
Jul 28 17:06:39.171311 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.171281    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-w64h8_cf8fb96f-7eea-4717-a0b7-5544121e1bdc/download-hf-model/0.log"
Jul 28 17:06:39.230479 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.230453    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-xcshw_b44cba6c-eef1-40f8-b4e7-071a7e2449de/s3-tls-init-serving/0.log"
Jul 28 17:06:39.270251 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.270218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-xcshw_b44cba6c-eef1-40f8-b4e7-071a7e2449de/download-hf-model/0.log"
Jul 28 17:06:39.359915 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:39.359882    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-nz4md_d1566c52-0860-4370-bb69-ae970a6f5d7f/seaweedfs-tls-custom/0.log"
Jul 28 17:06:45.750229 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.750197    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/kube-multus-additional-cni-plugins/0.log"
Jul 28 17:06:45.778895 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.778860    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/egress-router-binary-copy/0.log"
Jul 28 17:06:45.802311 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.802284    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/cni-plugins/0.log"
Jul 28 17:06:45.855485 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.855451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/bond-cni-plugin/0.log"
Jul 28 17:06:45.876557 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.876527    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/routeoverride-cni/0.log"
Jul 28 17:06:45.903559 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.903525    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/whereabouts-cni-bincopy/0.log"
Jul 28 17:06:45.932765 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.932733    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxwcf_3702206b-0647-4ea7-8cef-4fa6b7fa1fdd/whereabouts-cni/0.log"
Jul 28 17:06:45.987784 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:45.987742    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-drbzm_0dfe30ab-2aa2-416f-a1d8-f72f0767044a/kube-multus/0.log"
Jul 28 17:06:46.137831 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:46.137800    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krkl4_586a1d1d-6a0d-4cf6-9df0-41ee96f329a2/network-metrics-daemon/0.log"
Jul 28 17:06:46.158285 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:46.158255    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krkl4_586a1d1d-6a0d-4cf6-9df0-41ee96f329a2/kube-rbac-proxy/0.log"
Jul 28 17:06:46.994997 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:46.994964    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/ovn-controller/0.log"
Jul 28 17:06:47.019522 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.019494    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/ovn-acl-logging/0.log"
Jul 28 17:06:47.053279 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.053248    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/kube-rbac-proxy-node/0.log"
Jul 28 17:06:47.087297 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.087268    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 17:06:47.119279 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.119249    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/northd/0.log"
Jul 28 17:06:47.162516 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.162485    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/nbdb/0.log"
Jul 28 17:06:47.210866 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.210839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/sbdb/0.log"
Jul 28 17:06:47.323151 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:47.323030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2dc94_1eb13ef3-41ab-45ce-9a59-fc57643325f7/ovnkube-controller/0.log"
Jul 28 17:06:49.021334 ip-10-0-140-97 kubenswrapper[2580]: I0728 17:06:49.021301    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bnw82_3fa94af5-82f1-42a5-9a55-3de4251112e1/network-check-target-container/0.log"