Sep 18 19:14:26.036732 ip-10-0-135-205 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:14:29.578616 ip-10-0-135-205 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.578616 ip-10-0-135-205 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:14:29.578616 ip-10-0-135-205 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.578616 ip-10-0-135-205 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:14:29.578616 ip-10-0-135-205 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:14:29.579310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.578700    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:14:29.582640 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582612    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.582640 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582641    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.582640 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582646    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.582640 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582650    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582653    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582655    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582658    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582662    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582666    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582670    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582673    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582676    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582679    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582681    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582684    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582686    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582697    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582700    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582703    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582706    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582709    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582711    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582714    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.582764 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582716    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582719    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582722    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582724    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582727    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582729    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582732    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582735    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582738    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582740    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582743    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582745    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582747    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582750    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582754    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582757    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582760    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582762    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582765    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582767    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.583275 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582770    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582772    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582775    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582778    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582781    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582784    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582786    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582788    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582791    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582793    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582796    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582799    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582802    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582805    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582808    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582810    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582813    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582815    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582818    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582821    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.583820 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582824    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582826    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582829    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582831    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582834    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582836    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582841    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582844    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582848    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582852    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582855    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582858    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582861    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582863    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582866    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582868    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582871    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582873    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582876    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.584303 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582878    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582881    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582884    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582886    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582889    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582891    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582894    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582896    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582899    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582901    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582904    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582907    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582912    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582916    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582919    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582922    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.582925    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583644    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583651    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.584777 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583657    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583660    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583663    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583666    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583668    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583673    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583677    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583680    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583683    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583685    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583688    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583691    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583694    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583697    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583699    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583702    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583705    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583708    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583710    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583713    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.585234 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583715    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583718    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583720    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583723    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583726    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583728    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583732    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583735    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583737    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583740    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583742    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583745    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583747    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583750    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583752    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583754    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583757    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583760    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583762    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583765    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.585741 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583767    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583770    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583772    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583775    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583777    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583780    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583782    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583785    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583788    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583791    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583793    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583796    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583799    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583801    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583804    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583806    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583809    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583812    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583814    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583817    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.586251 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583820    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583823    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583826    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583830    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583834    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583837    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583840    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583842    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583845    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583848    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583851    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583854    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583858    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583861    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583863    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583866    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583869    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583872    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583874    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.586747 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583877    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583879    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583882    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583884    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583887    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583890    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583892    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583895    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583897    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583900    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583903    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583905    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583908    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583910    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583913    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583915    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583918    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.583920    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584296    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584301    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.587204 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584305    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584309    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584312    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584315    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584318    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584321    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584323    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584326    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584328    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584331    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584335    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584340    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584343    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584346    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584348    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584351    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584354    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584356    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584373    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584376    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.587715 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584379    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584385    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584388    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584390    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584392    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584395    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584398    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584401    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584403    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584406    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584409    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584412    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584414    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584417    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584419    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584423    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584425    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584428    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584431    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584433    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.588206 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584436    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584438    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584441    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584444    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584448    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584451    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584454    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584457    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584460    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584462    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584465    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584468    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584470    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584475    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584477    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584480    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584482    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584485    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584487    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584490    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.588709 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584492    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584494    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584497    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584500    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584502    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584505    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584507    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584510    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584513    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584515    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584518    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584520    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584523    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584525    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584528    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584531    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584534    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584537    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584539    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584542    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.589200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584545    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584547    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584550    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584552    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584555    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584559    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584562    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584564    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584567    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584569    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584572    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584574    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584577    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584579    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584582    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584584    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.584587    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584662    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584694    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584701    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:14:29.589706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584706    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584711    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584715    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584719    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584723    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584727    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584730    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584734    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584737    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584740    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584743    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584746    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584749    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584752    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584755    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584760    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584762    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584768    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584771    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584781    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584786    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584789    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584792    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:14:29.590189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584795    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584798    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584801    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584804    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584807    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584811    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584815    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584818    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584821    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584823    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584826    2583 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584835    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584839    2583 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584842    2583 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584845    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584848    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584851    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584857    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584860    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584863    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584866    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584869    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584872    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584875    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584877    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:14:29.590762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584880    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584884    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584887    2583 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584890    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584893    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584896    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584900    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584903    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584906    2583 flags.go:64] FLAG: --help="false"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584909    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584912    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584914    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584924    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584929    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584933    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584935    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584938    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584941    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584944    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584953    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584957    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584959    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584962    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584965    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:14:29.591432 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584969    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584972    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584975    2583 flags.go:64] FLAG: --lock-file=""
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584978    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584981    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584984    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584989    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584993    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584996    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.584999    2583 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585002    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585005    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585009    2583 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585011    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585016    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585019    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585024    2583 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585027    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585030    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585033    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585036    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585039    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585041    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585044    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585052    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:14:29.592013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585055    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585058    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585061    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585070    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585076    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585079    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585082    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585085    2583 flags.go:64] FLAG: --port="10250"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585089    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585092    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0919d636c82ce525f"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585095    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585098    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585101    2583 flags.go:64] FLAG: --register-node="true"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585104    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585108    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585111    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585114    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585116    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585120    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585123    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585126    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585129    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585132    2583 flags.go:64] FLAG: --runonce="false"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585135    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585138    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:14:29.592686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585140    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585143    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585146    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585149    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585152    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585155    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585158    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585161    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585164    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585167    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585170    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585181    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585184    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585189    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585192    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585196    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585203    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585206    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585209    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585212    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585215    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585218    2583 flags.go:64] FLAG: --v="2"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585222    2583 flags.go:64] FLAG: --version="false"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585226    2583 flags.go:64] FLAG: --vmodule=""
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585230    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:14:29.593297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585233    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585382    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585386    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585390    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585393    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585395    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585398    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585400    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585403    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585406    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585408    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585411    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585413    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585416    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585419    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585422    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585424    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585427    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585430    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585433    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.593917 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585435    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585438    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585443    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585446    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585449    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585452    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585455    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585457    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585460    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585462    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585465    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585467    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585470    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585473    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585475    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585478    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585482    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585486    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585489    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585492    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.594424 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585494    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585497    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585500    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585502    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585505    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585509    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585512    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585515    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585517    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585520    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585522    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585525    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585528    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585531    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585537    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585540    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585543    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585545    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585548    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.595084 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585551    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585553    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585556    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585558    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585561    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585564    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585567    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585569    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585572    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585574    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585577    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585580    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585582    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585585    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585588    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585590    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585593    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585595    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585598    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585601    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.595810 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585603    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585606    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585608    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585611    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585614    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585616    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585619    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585624    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585627    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585629    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585632    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585635    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585637    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585640    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585643    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585645    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585648    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585651    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585655    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.596317 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585658    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.585660    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.585666    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.592968    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.592986    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.593007    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.593013    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593078    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593083    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593087    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593090    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593093    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593096    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593099    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593102    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.596808 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593105    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593107    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593110    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593113    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593116    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593119    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593121    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593124    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593126    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593129    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593131    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593134    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593137    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593140    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593142    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593145    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593147    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593150    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593153    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.597213 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593155    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593158    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593162    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593165    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593168    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593170    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593173    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593176    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593179    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593181    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593184    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593187    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593189    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593192    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593194    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593197    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593199    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593202    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593205    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.597705 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593209    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593212    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593214    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593217    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593219    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593223    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593225    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593228    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593230    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593233    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593236    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593239    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593241    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593244    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593247    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593250    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593253    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593255    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593258    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593260    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.598182 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593263    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593265    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593269    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593273    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593276    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593279    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593282    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593285    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593288    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593290    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593293    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593296    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593299    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593303    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593306    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593308    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593311    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593315    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593319    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593322    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.598687 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593324    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593327    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593330    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593333    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593337    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593340    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593342    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593345    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593348    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593351    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593353    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593356    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593370    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.593376    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593498    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:14:29.599215 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593503    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593506    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593509    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593512    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593515    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593518    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593520    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593524    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593526    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593529    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593532    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593535    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593537    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593540    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593542    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593545    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593548    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593550    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593553    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593555    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:14:29.599600 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593558    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593561    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593563    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593566    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593569    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593571    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593574    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593578    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593582    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593586    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593589    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593592    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593595    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593598    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593600    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593603    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593606    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593608    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593611    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:14:29.600099 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593614    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593616    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593619    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593622    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593625    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593627    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593630    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593632    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593635    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593638    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593640    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593643    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593645    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593648    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593651    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593653    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593656    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593658    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593661    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593663    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:14:29.600815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593666    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593669    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593671    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593674    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593677    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593681    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593684    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593687    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593690    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593692    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593695    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593697    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593701    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593703    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593706    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593709    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593711    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593714    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593716    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593719    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:14:29.601299 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593721    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593725    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593728    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593731    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593734    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593736    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593739    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593742    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593744    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593747    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593749    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593752    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593754    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593757    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593760    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593763    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593765    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593768    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:14:29.601846 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:29.593770    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.593775    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.593909    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.596941    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.598013    2583 server.go:1019] "Starting client certificate rotation"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.598117    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.598155    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.600167    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:29.602289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.601385    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:14:29.613043 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.613026    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:14:29.614606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.614593    2583 log.go:25] "Validated CRI v1 image API"
Sep 18 19:14:29.616688 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.616665    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:14:29.618707 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.618688    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a19e9b62-2c8c-4bff-a55f-fb9cb30445f9:/dev/nvme0n1p3 e919d103-5495-40cc-96e9-81848ac787fc:/dev/nvme0n1p4]
Sep 18 19:14:29.618783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.618707    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:14:29.623840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.623718    2583 manager.go:216] Machine: {Timestamp:2026-09-18 19:14:29.622901843 +0000 UTC m=+1.998328137 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079465 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0a8d5b6342a108f50e8843dacb8d SystemUUID:ec2d0a8d-5b63-42a1-08f5-0e8843dacb8d BootID:aa56dbfa-ec68-4513-8b2e-d55726126748 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:5a:3d:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:5a:3d:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:5b:44:8a:a0:7c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:14:29.623840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.623834    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:14:29.623939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.623917    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:14:29.625594 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.625565    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:14:29.625737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.625595    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:14:29.626714 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.626701    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:14:29.626714 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.626715    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:14:29.626802 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.626727    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:29.626802 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.626738    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:14:29.626861 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.626813    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:14:29.629602 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629590    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:29.629725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629715    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:14:29.629761 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629751    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:14:29.629798 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629765    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:14:29.629798 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629783    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:14:29.629859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629799    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:14:29.629859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.629812    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:14:29.630294 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.630259    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:29.630400 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.630320    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:14:29.638456 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.638424    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:14:29.641909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.641887    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:14:29.642020 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.641922    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:14:29.643117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643098    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643124    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643134    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643142    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643156    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643164    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643179    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643184    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643192    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643201    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643213    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:14:29.643222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643221    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:14:29.643601 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643249    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:14:29.643601 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643258    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:14:29.643601 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.643404    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:29.644446 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.644422    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:14:29.644446 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.644426    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:14:29.647136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.647122    2583 server.go:1268] "Started kubelet"
Sep 18 19:14:29.647517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.647498    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:14:29.647820 ip-10-0-135-205 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:14:29.648450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.648437    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:14:29.649249 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.649191    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:14:29.649319 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.649278    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:14:29.650254 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.650234    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:14:29.653663 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.653636    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:14:29.654906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.654887    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:14:29.654906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.654897    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:29.655060 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.654969    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:29.655405 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.655384    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:14:29.655405 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.655387    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:14:29.655518 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.655413    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:14:29.655561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.655537    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:14:29.655561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.655544    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:14:29.657483 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.657459    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:14:29.657732 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.657694    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:14:29.657842 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.657721    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:14:29.657903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.657845    2583 factory.go:55] Registering systemd factory
Sep 18 19:14:29.657903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.657861    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:14:29.658141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658128    2583 factory.go:153] Registering CRI-O factory
Sep 18 19:14:29.658210 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658145    2583 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:14:29.658266 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658211    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:14:29.658266 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658231    2583 factory.go:103] Registering Raw factory
Sep 18 19:14:29.658266 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658250    2583 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:14:29.658796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.658699    2583 manager.go:349] Starting recovery of all containers
Sep 18 19:14:29.663376 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.663328    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:14:29.669978 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.669957    2583 manager.go:354] Recovery completed
Sep 18 19:14:29.684564 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.684541    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.685629 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.685610    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpggb"
Sep 18 19:14:29.686522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.686502    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.686582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.686537    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.686582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.686548    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.687112 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687096    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.687112 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687106    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:14:29.687238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687117    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:14:29.687238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687133    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:14:29.687887 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687870    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.687960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687901    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.687960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.687913    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.688545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.688532    2583 policy_none.go:49] "None policy: Start"
Sep 18 19:14:29.688610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.688551    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:14:29.688610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.688564    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:14:29.689995 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.689980    2583 policy_none.go:47] "Start"
Sep 18 19:14:29.699225 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.698247    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-205.ec2.internal.18d680359aeb848c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-205.ec2.internal,UID:ip-10-0-135-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-205.ec2.internal,},FirstTimestamp:2026-09-18 19:14:29.64709902 +0000 UTC m=+2.022525366,LastTimestamp:2026-09-18 19:14:29.64709902 +0000 UTC m=+2.022525366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-205.ec2.internal,}"
Sep 18 19:14:29.702912 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.702892    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpggb"
Sep 18 19:14:29.733073 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733050    2583 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:14:29.733280 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.733101    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:14:29.733280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733113    2583 server.go:87] "Starting device plugin registration server"
Sep 18 19:14:29.733456 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733443    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:14:29.733517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733469    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:14:29.733564 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733545    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:14:29.733648 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733634    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:14:29.733648 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.733647    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:14:29.734163 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.734134    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:14:29.734234 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.734193    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:29.785789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.785755    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:14:29.785931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.785817    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:14:29.785931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.785845    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:14:29.785931 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.785889    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:14:29.788951 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.788929    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:29.834157 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.834083    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.835167 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.835147    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.835220 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.835183    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.835220 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.835194    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.835291 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.835222    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.852874 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.852844    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.852959 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.852881    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-205.ec2.internal\": node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:29.886766 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.886726    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"]
Sep 18 19:14:29.886919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.886810    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.887803 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.887787    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.887860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.887820    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.887860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.887830    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.889055 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889043    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.889181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.889219 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889203    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.889858 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889840    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.889943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889868    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.889943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889881    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.889943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889893    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.889943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889920    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.889943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.889936    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.891590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.891575    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.891643 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.891602    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:29.892309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.892290    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:29.892417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.892324    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:29.892417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.892341    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:29.919219 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.919194    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-205.ec2.internal\" not found" node="ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.923722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.923703    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-205.ec2.internal\" not found" node="ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.938472 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:29.938448    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:29.957681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.957650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.957681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.957680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:29.957853 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:29.957698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91f5005043eff195cc3f3cd98b4b0f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"91f5005043eff195cc3f3cd98b4b0f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.039018 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.038982    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.058399 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.058494 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.058494 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91f5005043eff195cc3f3cd98b4b0f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"91f5005043eff195cc3f3cd98b4b0f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.058494 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91f5005043eff195cc3f3cd98b4b0f5f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-205.ec2.internal\" (UID: \"91f5005043eff195cc3f3cd98b4b0f5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.058590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.058590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.058507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6dec74abf6cf99519af3294786175e8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal\" (UID: \"e6dec74abf6cf99519af3294786175e8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.139650 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.139565    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.222598 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.222567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.227274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.226918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.240422 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.240395    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.340944 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.340904    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.441527 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.441441    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.542093 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.542054    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.597661 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.597629    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:14:30.598140 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.597814    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:14:30.622712 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.622684    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:30.642727 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.642571    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.707996 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.707903    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:09:29 +0000 UTC" deadline="2028-05-23 18:57:51.089627654 +0000 UTC"
Sep 18 19:14:30.707996 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.707952    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14711h43m20.381679429s"
Sep 18 19:14:30.716421 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.716391    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:14:30.717889 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.717870    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:14:30.718026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.717906    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:14:30.718026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.717920    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:14:30.743061 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.743036    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.819209 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:30.819168    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f5005043eff195cc3f3cd98b4b0f5f.slice/crio-36e81bd2b91f69050a52cb26f4f872713079a91ff455956fb2a8cefd32bba64a WatchSource:0}: Error finding container 36e81bd2b91f69050a52cb26f4f872713079a91ff455956fb2a8cefd32bba64a: Status 404 returned error can't find the container with id 36e81bd2b91f69050a52cb26f4f872713079a91ff455956fb2a8cefd32bba64a
Sep 18 19:14:30.819736 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:30.819708    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dec74abf6cf99519af3294786175e8.slice/crio-e5b0321c049393e1970d4b18fdd01704042ddda54b2bec3e36e56a7b37a2bd7d WatchSource:0}: Error finding container e5b0321c049393e1970d4b18fdd01704042ddda54b2bec3e36e56a7b37a2bd7d: Status 404 returned error can't find the container with id e5b0321c049393e1970d4b18fdd01704042ddda54b2bec3e36e56a7b37a2bd7d
Sep 18 19:14:30.824306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.824291    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:14:30.844116 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:30.844087    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-205.ec2.internal\" not found"
Sep 18 19:14:30.931013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.930984    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:30.955563 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.955532    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:30.983510 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.983457    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:30.985128 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:30.985112    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal"
Sep 18 19:14:31.004248 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.004226    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:14:31.114576 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.114544    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:31.631613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.631577    2583 apiserver.go:52] "Watching apiserver"
Sep 18 19:14:31.641548 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.641509    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:14:31.642010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.641986    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mqnsc","kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf","openshift-cluster-node-tuning-operator/tuned-wtz8z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal","openshift-multus/multus-additional-cni-plugins-wz8cl","openshift-multus/multus-ftftt","openshift-network-diagnostics/network-check-target-xz56f","kube-system/konnectivity-agent-zhdrh","openshift-dns/node-resolver-4zxvd","openshift-image-registry/node-ca-xh8cv","openshift-multus/network-metrics-daemon-nsgdm","openshift-network-operator/iptables-alerter-5wsf8","openshift-ovn-kubernetes/ovnkube-node-7fbdn"]
Sep 18 19:14:31.643808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.643785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.645162 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.645122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.646317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.646296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.647549 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.647529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:31.647656 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.647617    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:31.648843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.648822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.650398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.650379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.651093 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.651067    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.652345 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.652180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.652345 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.652253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:31.653225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653056    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.653225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653121    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.653225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdzzp\""
Sep 18 19:14:31.653225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:14:31.653997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.653997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653699    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:14:31.653997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w8tt6\""
Sep 18 19:14:31.653997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.653897    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:14:31.654227 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qmdbr\""
Sep 18 19:14:31.654279 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654251    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.654522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654506    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.654522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654516    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:14:31.654677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654539    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:14:31.654923 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654908    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.654983 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.654926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l5nsb\""
Sep 18 19:14:31.655260 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.655236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.655814 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.655791    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:14:31.656017 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.655999    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b7s89\""
Sep 18 19:14:31.657166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.656812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.658124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.658106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.658224 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.658163    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:31.658596 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.658575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:14:31.658693 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.658583    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:14:31.659400 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.659352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.660845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.660768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:14:31.660845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.660791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.661009 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.660894    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:14:31.661009 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.660926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8s6sd\""
Sep 18 19:14:31.661009 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.660975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.661195 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661181    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.661268 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661182    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9prhf\""
Sep 18 19:14:31.661318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661185    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.661318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.661435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kfc8q\""
Sep 18 19:14:31.661543 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.661523    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.664810 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.664778    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fm96q\""
Sep 18 19:14:31.666486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.666462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:14:31.666660 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.666541    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:14:31.667227 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-daemon-config\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.667318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cnibin\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.667318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xn5\" (UniqueName: \"kubernetes.io/projected/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-kube-api-access-s5xn5\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.667318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-serviceca\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-modprobe-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-sys\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-kubelet\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-run\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.667508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-dbus\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.667809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.667960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysconfig\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.668021 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.667988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-device-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.668078 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-conf\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.668078 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-netns\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668183 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-conf-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668183 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpjr\" (UniqueName: \"kubernetes.io/projected/d1ec9de2-8739-4f3b-b00c-cb71176631f2-kube-api-access-qvpjr\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668183 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b2692c8-adc0-4005-8214-5c90e6e12d4f-agent-certs\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.668183 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b2692c8-adc0-4005-8214-5c90e6e12d4f-konnectivity-ca\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.668378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-socket-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.668378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/561e9854-ab29-47f0-b3e0-620407e4d964-hosts-file\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.668378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-host-slash\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.668378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.668545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-system-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cnibin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-hostroot\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.668680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.668680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblhh\" (UniqueName: \"kubernetes.io/projected/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-kube-api-access-sblhh\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.668680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:14:31.670143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.668687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:31.671449 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cni-binary-copy\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671560 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5jc\" (UniqueName: \"kubernetes.io/projected/561e9854-ab29-47f0-b3e0-620407e4d964-kube-api-access-jn5jc\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.671560 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5tw\" (UniqueName: \"kubernetes.io/projected/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-kube-api-access-lm5tw\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.671560 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.671711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-lib-modules\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.671711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-host\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.671711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-multus\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-host\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.671711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-k8s-cni-cncf-io\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-multus-certs\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-sys-fs\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm687\" (UniqueName: \"kubernetes.io/projected/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-kube-api-access-zm687\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-etc-kubernetes\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.671940 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-kubelet-config\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561e9854-ab29-47f0-b3e0-620407e4d964-tmp-dir\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.671993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-bin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-tuned\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.672272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-socket-dir-parent\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.672631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9np\" (UniqueName: \"kubernetes.io/projected/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kube-api-access-nz9np\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.672631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-registration-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.672631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-var-lib-kubelet\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.673095 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-tmp\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.674033 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.673999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-iptables-alerter-script\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.674114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.674047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.674114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.674074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-kubernetes\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.674114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.674098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-systemd\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.674255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672818    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:14:31.674302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.672950    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:14:31.677831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.677709    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:14:31.718931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.718892    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q94mw"
Sep 18 19:14:31.735103 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.735070    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q94mw"
Sep 18 19:14:31.756347 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.756314    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:14:31.775094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-kubernetes\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-systemd\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-slash\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-kubernetes\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/470b627f-f8be-4a46-9b20-6013f070ba92-ovn-node-metrics-cert\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775305    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-systemd\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-daemon-config\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cnibin\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xn5\" (UniqueName: \"kubernetes.io/projected/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-kube-api-access-s5xn5\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cnibin\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-serviceca\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-modprobe-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-sys\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.775604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-kubelet\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-sys\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-run\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-modprobe-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-dbus\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-kubelet\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysconfig\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-ovn\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-run\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-device-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysconfig\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-dbus\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775841    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-conf\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-netns\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-device-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-conf-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-netns\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-conf\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpjr\" (UniqueName: \"kubernetes.io/projected/d1ec9de2-8739-4f3b-b00c-cb71176631f2-kube-api-access-qvpjr\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-conf-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b2692c8-adc0-4005-8214-5c90e6e12d4f-agent-certs\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b2692c8-adc0-4005-8214-5c90e6e12d4f-konnectivity-ca\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-socket-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.775994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-serviceca\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/561e9854-ab29-47f0-b3e0-620407e4d964-hosts-file\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-daemon-config\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-host-slash\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-host-slash\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-system-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-system-cni-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/561e9854-ab29-47f0-b3e0-620407e4d964-hosts-file\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cnibin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-hostroot\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-system-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.776974 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-socket-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblhh\" (UniqueName: \"kubernetes.io/projected/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-kube-api-access-sblhh\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cnibin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-binary-copy\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776331    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-hostroot\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776387    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cni-binary-copy\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5jc\" (UniqueName: \"kubernetes.io/projected/561e9854-ab29-47f0-b3e0-620407e4d964-kube-api-access-jn5jc\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkwq\" (UniqueName: \"kubernetes.io/projected/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-kube-api-access-qdkwq\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-systemd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-log-socket\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-bin\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jtz\" (UniqueName: \"kubernetes.io/projected/470b627f-f8be-4a46-9b20-6013f070ba92-kube-api-access-22jtz\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5tw\" (UniqueName: \"kubernetes.io/projected/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-kube-api-access-lm5tw\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.777815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-lib-modules\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-host\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-env-overrides\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-multus\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-host\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-k8s-cni-cncf-io\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-host\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-multus-certs\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-multus\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-sys-fs\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-multus-certs\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1ec9de2-8739-4f3b-b00c-cb71176631f2-cni-binary-copy\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-sys-fs\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.776865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-lib-modules\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm687\" (UniqueName: \"kubernetes.io/projected/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-kube-api-access-zm687\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.779506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-cni-dir\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b2692c8-adc0-4005-8214-5c90e6e12d4f-konnectivity-ca\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-run-k8s-cni-cncf-io\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-systemd-units\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-host\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-config\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-etc-kubernetes\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-kubelet-config\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561e9854-ab29-47f0-b3e0-620407e4d964-tmp-dir\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-kubelet-config\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-netns\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-bin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.777320    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-tuned\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-netd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.780275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-script-lib\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-socket-dir-parent\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9np\" (UniqueName: \"kubernetes.io/projected/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kube-api-access-nz9np\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-etc-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-registration-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/561e9854-ab29-47f0-b3e0-620407e4d964-tmp-dir\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.777572    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.277550677 +0000 UTC m=+4.652976956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-registration-dir\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-var-lib-kubelet\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-tmp\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-host-var-lib-cni-bin\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-kubelet\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-var-lib-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-node-log\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-multus-socket-dir-parent\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-iptables-alerter-script\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-var-lib-kubelet\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-sysctl-d\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.777236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ec9de2-8739-4f3b-b00c-cb71176631f2-etc-kubernetes\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.778224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-iptables-alerter-script\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.780385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-etc-tuned\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.780605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b2692c8-adc0-4005-8214-5c90e6e12d4f-agent-certs\") pod \"konnectivity-agent-zhdrh\" (UID: \"0b2692c8-adc0-4005-8214-5c90e6e12d4f\") " pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.781565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.780842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-tmp\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.789484 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.789452    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:31.789484 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.789477    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:31.789661 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.789491    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:31.789661 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.789564    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.289545435 +0000 UTC m=+4.664971725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:31.790900 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.790851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" event={"ID":"91f5005043eff195cc3f3cd98b4b0f5f","Type":"ContainerStarted","Data":"36e81bd2b91f69050a52cb26f4f872713079a91ff455956fb2a8cefd32bba64a"}
Sep 18 19:14:31.791662 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.791643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblhh\" (UniqueName: \"kubernetes.io/projected/faf1d939-4798-4480-bb3d-ba0f6cd73c6a-kube-api-access-sblhh\") pod \"node-ca-xh8cv\" (UID: \"faf1d939-4798-4480-bb3d-ba0f6cd73c6a\") " pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:31.792236 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.792210    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"e6dec74abf6cf99519af3294786175e8","Type":"ContainerStarted","Data":"e5b0321c049393e1970d4b18fdd01704042ddda54b2bec3e36e56a7b37a2bd7d"}
Sep 18 19:14:31.794656 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.794625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm687\" (UniqueName: \"kubernetes.io/projected/d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78-kube-api-access-zm687\") pod \"tuned-wtz8z\" (UID: \"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78\") " pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.794934 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.794912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9np\" (UniqueName: \"kubernetes.io/projected/4a4b948f-0396-4a7f-a2a4-8a2cc7803bda-kube-api-access-nz9np\") pod \"aws-ebs-csi-driver-node-vjdcf\" (UID: \"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:31.795674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.795649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5tw\" (UniqueName: \"kubernetes.io/projected/a3b5cc85-257b-42a2-834b-ef7adbc2ad99-kube-api-access-lm5tw\") pod \"iptables-alerter-5wsf8\" (UID: \"a3b5cc85-257b-42a2-834b-ef7adbc2ad99\") " pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.796064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.796039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xn5\" (UniqueName: \"kubernetes.io/projected/6a905a6d-caab-41dd-8d3d-955f1ff08b0b-kube-api-access-s5xn5\") pod \"multus-additional-cni-plugins-wz8cl\" (UID: \"6a905a6d-caab-41dd-8d3d-955f1ff08b0b\") " pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.796868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.796846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpjr\" (UniqueName: \"kubernetes.io/projected/d1ec9de2-8739-4f3b-b00c-cb71176631f2-kube-api-access-qvpjr\") pod \"multus-ftftt\" (UID: \"d1ec9de2-8739-4f3b-b00c-cb71176631f2\") " pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.797500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.797482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5jc\" (UniqueName: \"kubernetes.io/projected/561e9854-ab29-47f0-b3e0-620407e4d964-kube-api-access-jn5jc\") pod \"node-resolver-4zxvd\" (UID: \"561e9854-ab29-47f0-b3e0-620407e4d964\") " pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:31.878942 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.878896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-systemd-units\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.878951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-config\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.878996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-netns\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-netd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-systemd-units\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-netd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-script-lib\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-netns\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-etc-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-kubelet\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-var-lib-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-node-log\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-etc-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-kubelet\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-slash\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-var-lib-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/470b627f-f8be-4a46-9b20-6013f070ba92-ovn-node-metrics-cert\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-node-log\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-openvswitch\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-slash\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-ovn\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.879433    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:31.879914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkwq\" (UniqueName: \"kubernetes.io/projected/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-kube-api-access-qdkwq\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-ovn\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-systemd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:31.879506    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:32.379486518 +0000 UTC m=+4.754912797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-run-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-run-systemd\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-log-socket\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-bin\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jtz\" (UniqueName: \"kubernetes.io/projected/470b627f-f8be-4a46-9b20-6013f070ba92-kube-api-access-22jtz\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-config\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-log-socket\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879665    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-cni-bin\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-env-overrides\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/470b627f-f8be-4a46-9b20-6013f070ba92-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-ovnkube-script-lib\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.880785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.879974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/470b627f-f8be-4a46-9b20-6013f070ba92-env-overrides\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.881888 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.881834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/470b627f-f8be-4a46-9b20-6013f070ba92-ovn-node-metrics-cert\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.891799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.891762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkwq\" (UniqueName: \"kubernetes.io/projected/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-kube-api-access-qdkwq\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:31.892855 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.892826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jtz\" (UniqueName: \"kubernetes.io/projected/470b627f-f8be-4a46-9b20-6013f070ba92-kube-api-access-22jtz\") pod \"ovnkube-node-7fbdn\" (UID: \"470b627f-f8be-4a46-9b20-6013f070ba92\") " pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:31.955747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.955715    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:14:31.959676 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.959648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftftt"
Sep 18 19:14:31.966909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.966887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wsf8"
Sep 18 19:14:31.978941 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.978917    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:31.984738 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.984715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtz8z"
Sep 18 19:14:31.993613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.993590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wz8cl"
Sep 18 19:14:31.999239 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:31.999219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf"
Sep 18 19:14:32.006176 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.006153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4zxvd"
Sep 18 19:14:32.013742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.013718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xh8cv"
Sep 18 19:14:32.019836 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.019811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:32.282623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.282586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:32.282788 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.282747    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:32.282837 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.282808    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.282791578 +0000 UTC m=+5.658217866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:32.383580 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.383542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:32.383737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.383605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:32.383737 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383698    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:32.383737 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383730    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:32.383892 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383749    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:32.383892 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383761    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:32.383892 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.383759305 +0000 UTC m=+5.759185575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:32.383892 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.383805    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:33.383791976 +0000 UTC m=+5.759218248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:32.660684 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.660658    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a521ca_d5c8_46d4_9cd5_cbb39b1cfe78.slice/crio-6e51e2af99349b6c1dff76f447805463f5c296d0a09ba775b2ff440807cb1507 WatchSource:0}: Error finding container 6e51e2af99349b6c1dff76f447805463f5c296d0a09ba775b2ff440807cb1507: Status 404 returned error can't find the container with id 6e51e2af99349b6c1dff76f447805463f5c296d0a09ba775b2ff440807cb1507
Sep 18 19:14:32.662010 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.661974    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470b627f_f8be_4a46_9b20_6013f070ba92.slice/crio-e68224193cab178c5a9c37129f56d7cc279a402cfc418351270dc0b585a1e4d3 WatchSource:0}: Error finding container e68224193cab178c5a9c37129f56d7cc279a402cfc418351270dc0b585a1e4d3: Status 404 returned error can't find the container with id e68224193cab178c5a9c37129f56d7cc279a402cfc418351270dc0b585a1e4d3
Sep 18 19:14:32.665984 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.665945    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b5cc85_257b_42a2_834b_ef7adbc2ad99.slice/crio-e570729618be4b45794d5cf0e759dd7131ee9660f3bde223e62a50eeb80981e5 WatchSource:0}: Error finding container e570729618be4b45794d5cf0e759dd7131ee9660f3bde223e62a50eeb80981e5: Status 404 returned error can't find the container with id e570729618be4b45794d5cf0e759dd7131ee9660f3bde223e62a50eeb80981e5
Sep 18 19:14:32.667979 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.667957    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ec9de2_8739_4f3b_b00c_cb71176631f2.slice/crio-f9eed33d74b0dd20413a87effa66881c2f39e6023d1efeed69c43273177850dc WatchSource:0}: Error finding container f9eed33d74b0dd20413a87effa66881c2f39e6023d1efeed69c43273177850dc: Status 404 returned error can't find the container with id f9eed33d74b0dd20413a87effa66881c2f39e6023d1efeed69c43273177850dc
Sep 18 19:14:32.668867 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.668837    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a905a6d_caab_41dd_8d3d_955f1ff08b0b.slice/crio-c8eee2234417a0678db1bef7fc849829758bb8b434260ed3a8d853fff325c06e WatchSource:0}: Error finding container c8eee2234417a0678db1bef7fc849829758bb8b434260ed3a8d853fff325c06e: Status 404 returned error can't find the container with id c8eee2234417a0678db1bef7fc849829758bb8b434260ed3a8d853fff325c06e
Sep 18 19:14:32.670568 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:14:32.669478    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf1d939_4798_4480_bb3d_ba0f6cd73c6a.slice/crio-724bb14872e6688af737248bcd808241960fedea1395fe235e3d91cd466f17d5 WatchSource:0}: Error finding container 724bb14872e6688af737248bcd808241960fedea1395fe235e3d91cd466f17d5: Status 404 returned error can't find the container with id 724bb14872e6688af737248bcd808241960fedea1395fe235e3d91cd466f17d5
Sep 18 19:14:32.737478 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.737432    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:31 +0000 UTC" deadline="2028-06-22 12:22:38.04849058 +0000 UTC"
Sep 18 19:14:32.737478 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.737474    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15425h8m5.311019s"
Sep 18 19:14:32.786465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.786432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:32.786636 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:32.786558    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:32.797641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.797604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" event={"ID":"91f5005043eff195cc3f3cd98b4b0f5f","Type":"ContainerStarted","Data":"a2252c972b1d69811554063ca160603e0c96e6f3fdc5de6242c2acf5d599bc31"}
Sep 18 19:14:32.799145 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.799117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zxvd" event={"ID":"561e9854-ab29-47f0-b3e0-620407e4d964","Type":"ContainerStarted","Data":"6db6a2916c24a333fb6cc7491783aec12e15376257b463547ad8ae34d13c8414"}
Sep 18 19:14:32.800318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.800295    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhdrh" event={"ID":"0b2692c8-adc0-4005-8214-5c90e6e12d4f","Type":"ContainerStarted","Data":"f7797bb0a29cd6cd070cf2e5bc78d07456efecdc6e638301c6c6c2dbe1c07f88"}
Sep 18 19:14:32.801737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.801716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf" event={"ID":"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda","Type":"ContainerStarted","Data":"36c41c4cde02279ea2e9954e3436e1cfb002c55e7a2f035de8fc6445dd969160"}
Sep 18 19:14:32.802841 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.802820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerStarted","Data":"c8eee2234417a0678db1bef7fc849829758bb8b434260ed3a8d853fff325c06e"}
Sep 18 19:14:32.803967 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.803936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftftt" event={"ID":"d1ec9de2-8739-4f3b-b00c-cb71176631f2","Type":"ContainerStarted","Data":"f9eed33d74b0dd20413a87effa66881c2f39e6023d1efeed69c43273177850dc"}
Sep 18 19:14:32.805792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.805770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xh8cv" event={"ID":"faf1d939-4798-4480-bb3d-ba0f6cd73c6a","Type":"ContainerStarted","Data":"724bb14872e6688af737248bcd808241960fedea1395fe235e3d91cd466f17d5"}
Sep 18 19:14:32.806854 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.806834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wsf8" event={"ID":"a3b5cc85-257b-42a2-834b-ef7adbc2ad99","Type":"ContainerStarted","Data":"e570729618be4b45794d5cf0e759dd7131ee9660f3bde223e62a50eeb80981e5"}
Sep 18 19:14:32.807910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.807890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"e68224193cab178c5a9c37129f56d7cc279a402cfc418351270dc0b585a1e4d3"}
Sep 18 19:14:32.809749 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:32.809731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtz8z" event={"ID":"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78","Type":"ContainerStarted","Data":"6e51e2af99349b6c1dff76f447805463f5c296d0a09ba775b2ff440807cb1507"}
Sep 18 19:14:33.288770 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.288537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:33.288975 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.288826    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:33.288975 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.288904    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.288884605 +0000 UTC m=+7.664310891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:33.389899 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.389862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:33.390064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.389960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:33.390137 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390054    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:33.390137 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390077    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:33.390137 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390088    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:33.390137 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390102    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:33.390333 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390143    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.390125254 +0000 UTC m=+7.765551528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:33.390333 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.390167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:35.390150365 +0000 UTC m=+7.765576653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:33.737891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.737802    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:09:31 +0000 UTC" deadline="2028-05-03 04:51:44.881704515 +0000 UTC"
Sep 18 19:14:33.737891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.737840    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14217h37m11.14386868s"
Sep 18 19:14:33.787022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.786986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:33.787192 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.787123    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:33.787761 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.787618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:33.787761 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:33.787719    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:33.820312 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.820278    2583 generic.go:356] "Generic (PLEG): container finished" podID="e6dec74abf6cf99519af3294786175e8" containerID="e8a37f681648e5c244de9d724883a60f47af74a48564cf994b5b22fcfcab4be3" exitCode=0
Sep 18 19:14:33.820489 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.820396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"e6dec74abf6cf99519af3294786175e8","Type":"ContainerDied","Data":"e8a37f681648e5c244de9d724883a60f47af74a48564cf994b5b22fcfcab4be3"}
Sep 18 19:14:33.865417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:33.865331    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-205.ec2.internal" podStartSLOduration=3.865308251 podStartE2EDuration="3.865308251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:32.835941638 +0000 UTC m=+5.211367933" watchObservedRunningTime="2026-09-18 19:14:33.865308251 +0000 UTC m=+6.240734547"
Sep 18 19:14:34.786720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:34.786179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:34.786720 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:34.786332    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:34.843339 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:34.842595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" event={"ID":"e6dec74abf6cf99519af3294786175e8","Type":"ContainerStarted","Data":"c243bdd5bb3adbc9770a9c39bc5100a96e95b72a41741ee0a93f127c3d29376a"}
Sep 18 19:14:35.308050 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:35.307659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:35.308050 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.307843    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:35.308050 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.307910    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:39.307892062 +0000 UTC m=+11.683318357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:35.408511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:35.408466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:35.408695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:35.408538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:35.409063 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.409025    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:35.409155 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.409074    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:35.409155 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.409103    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:35.409261 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.409189    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:39.409156401 +0000 UTC m=+11.784582695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:35.419852 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.419534    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:35.419852 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.419628    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:39.419605009 +0000 UTC m=+11.795031284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:35.787998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:35.787480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:35.787998 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.787613    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:35.787998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:35.787809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:35.787998 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:35.787929    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:36.786777 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:36.786714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:36.786946 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:36.786876    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:37.786561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:37.786528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:37.786561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:37.786562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:37.787043 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:37.786657    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:37.787089 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:37.787052    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:38.786664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:38.786605    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:38.787128 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:38.786763    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:39.343559 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:39.343512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:39.343764 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.343681    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:39.343764 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.343753    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:47.343733785 +0000 UTC m=+19.719160069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:39.444238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:39.444193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:39.444440 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:39.444263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:39.444533 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444455    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:39.444533 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444474    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:39.444533 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444489    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:39.444682 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:47.444532156 +0000 UTC m=+19.819958440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:39.444682 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444621    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:39.444682 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.444654    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:14:47.444642512 +0000 UTC m=+19.820068790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:39.786344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:39.786296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:39.787573 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.787530    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:39.787996 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:39.787641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:39.787996 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:39.787758    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:40.787037 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:40.786986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:40.787221 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:40.787124    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:41.786712 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:41.786674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:41.787138 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:41.786813    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:41.787138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:41.786897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:41.787138 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:41.787032    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:42.786604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:42.786568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:42.786769 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:42.786707    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:43.787119 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:43.787084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:43.787570 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:43.787084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:43.787570 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:43.787211    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:43.787570 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:43.787269    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:44.786633 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:44.786597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:44.786796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:44.786751    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:45.789931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:45.789899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:45.790332 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:45.790021    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:45.790332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:45.790075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:45.790332 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:45.790181    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:46.786537 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:46.786500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:46.786733 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:46.786631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:47.406053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:47.406006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:47.406573 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.406180    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:47.406573 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.406269    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:03.406244397 +0000 UTC m=+35.781670745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:14:47.507045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:47.507001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:47.507070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507154    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507175    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507190    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507204    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:47.507235 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507235    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:03.507213728 +0000 UTC m=+35.882640016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:14:47.507550 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.507255    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:03.507245383 +0000 UTC m=+35.882671667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:14:47.787018 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:47.786985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:47.787018 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:47.787003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:47.787233 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.787107    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:47.787233 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:47.787201    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:48.786416 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:48.786381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:48.786862 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:48.786516    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:49.787564 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:49.787524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:49.787966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:49.787606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:49.787966 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:49.787616    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:49.787966 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:49.787722    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:50.786720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.786535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:50.786958 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:50.786830    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:50.881943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.881675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftftt" event={"ID":"d1ec9de2-8739-4f3b-b00c-cb71176631f2","Type":"ContainerStarted","Data":"1d5e66ee143991db6856136fb4616ab01ea15871c0e6f70cdb3dd9eccf98a488"}
Sep 18 19:14:50.883688 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.883657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xh8cv" event={"ID":"faf1d939-4798-4480-bb3d-ba0f6cd73c6a","Type":"ContainerStarted","Data":"88e2d6deb588df52be165a46cbdec3b0ffb739c2f35992944feb306edadf5e9b"}
Sep 18 19:14:50.887273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"41ad93a0eb67bc1a12a0a23b31a2ba94e287e7bcf29feedc70c709f9aaf73447"}
Sep 18 19:14:50.887398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"76bc7ff8a21e866266094c9dc27fb3b1805b43cec0727346a908dd5599a57376"}
Sep 18 19:14:50.887398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"5240b641a3fd254807acfa0cc96facd80f9b12f01ccfd940b61bca12a055adcb"}
Sep 18 19:14:50.887506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"1fe26d9dd38cde2a2cde8e6f6d3f96b86cb8929034174188af0bac0cfe6726da"}
Sep 18 19:14:50.887506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"bcdac65f124a7d90bcf2ad9b97e4d854ed35007b18b5b80d156905eeec03be32"}
Sep 18 19:14:50.887506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.887501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"7b528dd1e41f9e3a1fe684b2a6fcb78c29425b994c5e4020d3ab5daf654fd9e7"}
Sep 18 19:14:50.889043 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.889018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtz8z" event={"ID":"d4a521ca-d5c8-46d4-9cd5-cbb39b1cfe78","Type":"ContainerStarted","Data":"8e9c9d502ed3e163c9991e50b35c7b969f0d3d33f443ce87385fa81223685080"}
Sep 18 19:14:50.890728 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.890701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4zxvd" event={"ID":"561e9854-ab29-47f0-b3e0-620407e4d964","Type":"ContainerStarted","Data":"d7657042fc3895610aee3a9bda715366f067f1d1ed447b7b301faf3f70f9d133"}
Sep 18 19:14:50.892454 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.892430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhdrh" event={"ID":"0b2692c8-adc0-4005-8214-5c90e6e12d4f","Type":"ContainerStarted","Data":"00b2ccf60b84c2e7d7067b31b8f6cd616a6df4383f9c0e120ef4e949af6043fb"}
Sep 18 19:14:50.894143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.894120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf" event={"ID":"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda","Type":"ContainerStarted","Data":"405520413b7c5a0166e6b50d5773fe596ab971958c662565ec8ba5b1540d617d"}
Sep 18 19:14:50.895843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.895815    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="d4e967239ab070f198044fd54e8b3bab928005cc0b24e9d584826d63242e00ff" exitCode=0
Sep 18 19:14:50.895939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.895853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"d4e967239ab070f198044fd54e8b3bab928005cc0b24e9d584826d63242e00ff"}
Sep 18 19:14:50.924730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.924678    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-205.ec2.internal" podStartSLOduration=20.924662272 podStartE2EDuration="20.924662272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:34.906382541 +0000 UTC m=+7.281808836" watchObservedRunningTime="2026-09-18 19:14:50.924662272 +0000 UTC m=+23.300088566"
Sep 18 19:14:50.957629 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.957577    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xh8cv" podStartSLOduration=3.647264447 podStartE2EDuration="20.957560523s" totalImagesPullingTime="17.310296076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:50.95689512 +0000 UTC m=+23.332321413" watchObservedRunningTime="2026-09-18 19:14:50.957560523 +0000 UTC m=+23.332986815"
Sep 18 19:14:50.958064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:50.958041    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ftftt" podStartSLOduration=4.598573913 podStartE2EDuration="21.95803482s" totalImagesPullingTime="17.359460907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:50.928334832 +0000 UTC m=+23.303761138" watchObservedRunningTime="2026-09-18 19:14:50.95803482 +0000 UTC m=+23.333461131"
Sep 18 19:14:51.001532 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.001480    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zhdrh" podStartSLOduration=12.108344206 podStartE2EDuration="22.001468153s" totalImagesPullingTime="9.893123947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.000954065 +0000 UTC m=+23.376380378" watchObservedRunningTime="2026-09-18 19:14:51.001468153 +0000 UTC m=+23.376894445"
Sep 18 19:14:51.088604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.088564    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wtz8z" podStartSLOduration=3.7659925039999997 podStartE2EDuration="21.088547995s" totalImagesPullingTime="17.322555491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.087306375 +0000 UTC m=+23.462732668" watchObservedRunningTime="2026-09-18 19:14:51.088547995 +0000 UTC m=+23.463974287"
Sep 18 19:14:51.114933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.114907    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:14:51.746246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.746079    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:14:51.114927736Z","UUID":"72c79d57-1a19-476a-9ea3-ba82b03018bc","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:14:51.747807 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.747784    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:14:51.747951 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.747819    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:14:51.786914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.786852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:51.787093 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:51.786983    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:51.787390 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.787344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:51.787493 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:51.787465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:51.900863 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.900820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf" event={"ID":"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda","Type":"ContainerStarted","Data":"411bc65bfe7802ca31bb5738feecdf21469eb58c5038ad35bcaa9050cf14f1a4"}
Sep 18 19:14:51.903753 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.903721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wsf8" event={"ID":"a3b5cc85-257b-42a2-834b-ef7adbc2ad99","Type":"ContainerStarted","Data":"f7f2191a96ccef5f589982fd022fd32041b8f934bae4817f591db7f42dec0772"}
Sep 18 19:14:51.937737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:51.937592    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4zxvd" podStartSLOduration=4.631363607 podStartE2EDuration="21.937570194s" totalImagesPullingTime="17.306206587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.128112269 +0000 UTC m=+23.503538573" watchObservedRunningTime="2026-09-18 19:14:51.937570194 +0000 UTC m=+24.312996488"
Sep 18 19:14:52.786170 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:52.786131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:52.786420 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:52.786302    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:52.909614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:52.909523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"5ad34ad1f829572f8f764b045db83c77cfbe63b9c024bbb26dc113c9fae7c78c"}
Sep 18 19:14:52.912157 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:52.912122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf" event={"ID":"4a4b948f-0396-4a7f-a2a4-8a2cc7803bda","Type":"ContainerStarted","Data":"8d9beca5d59b96eba3adc376c6a24c8c1a839f7b2825d32b93814564998771c8"}
Sep 18 19:14:52.946907 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:52.946858    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5wsf8" podStartSLOduration=6.62967614 podStartE2EDuration="23.946840952s" totalImagesPullingTime="17.317164812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:51.937714191 +0000 UTC m=+24.313140484" watchObservedRunningTime="2026-09-18 19:14:52.946840952 +0000 UTC m=+25.322267244"
Sep 18 19:14:53.062047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.062011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:53.062736 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.062712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:53.089721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.089665    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjdcf" podStartSLOduration=4.241954794 podStartE2EDuration="23.089644358s" totalImagesPullingTime="18.847689564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:52.947504341 +0000 UTC m=+25.322930634" watchObservedRunningTime="2026-09-18 19:14:53.089644358 +0000 UTC m=+25.465070727"
Sep 18 19:14:53.790388 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.790343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:53.790528 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:53.790486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:53.790600 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.790544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:53.790722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:53.790699    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:53.915234 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.915197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:53.915705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:53.915687    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zhdrh"
Sep 18 19:14:54.786178 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:54.786142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:54.786357 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:54.786281    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:55.786789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:55.786619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:55.787495 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:55.786680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:55.787495 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:55.786879    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:55.787495 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:55.786927    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:55.921707 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:55.921673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerStarted","Data":"e450b47542144171b5e903e29f08f9a4753a85a9fe721005d82f369245cc2544"}
Sep 18 19:14:55.925352 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:55.925323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" event={"ID":"470b627f-f8be-4a46-9b20-6013f070ba92","Type":"ContainerStarted","Data":"8e5ba6091f93e6002ff7d015e2a3959a48314162c1e3d3b5f50214384ac93a7b"}
Sep 18 19:14:56.041149 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.041098    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn" podStartSLOduration=8.690130647 podStartE2EDuration="26.041081314s" totalImagesPullingTime="17.350950667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:14:56.039145155 +0000 UTC m=+28.414571441" watchObservedRunningTime="2026-09-18 19:14:56.041081314 +0000 UTC m=+28.416507606"
Sep 18 19:14:56.786587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.786549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:56.786736 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:56.786666    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:56.929096 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.929060    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="e450b47542144171b5e903e29f08f9a4753a85a9fe721005d82f369245cc2544" exitCode=0
Sep 18 19:14:56.929528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.929110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"e450b47542144171b5e903e29f08f9a4753a85a9fe721005d82f369245cc2544"}
Sep 18 19:14:56.929729 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.929709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:56.929821 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.929788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:56.930140 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.929839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:56.944961 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.944933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:56.945236 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:56.945222    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:14:57.584623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.584426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xz56f"]
Sep 18 19:14:57.584891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.584736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:57.584891 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:57.584857    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:57.588948 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.588807    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqnsc"]
Sep 18 19:14:57.588948 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.588931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:57.589132 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:57.589034    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:57.599784 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.599752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsgdm"]
Sep 18 19:14:57.599922 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:57.599867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:57.600068 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:57.599963    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:58.786081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:58.786052    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:14:58.786424 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:58.786165    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:14:58.938022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:58.935775    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="b172118c948811e7395a247d0b51d966eaf300c4bb1aec5bcb81a1950f79d81d" exitCode=0
Sep 18 19:14:58.938022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:58.936545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"b172118c948811e7395a247d0b51d966eaf300c4bb1aec5bcb81a1950f79d81d"}
Sep 18 19:14:59.787332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:59.787300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:14:59.787670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:59.787394    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:14:59.787670 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:59.787411    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:14:59.787670 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:14:59.787509    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:14:59.940852 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:59.940745    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="d0a1e8a764f2b5ab85169822f1270445656165749ba2e678ce7f3a6e607e96d8" exitCode=0
Sep 18 19:14:59.940852 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:14:59.940826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"d0a1e8a764f2b5ab85169822f1270445656165749ba2e678ce7f3a6e607e96d8"}
Sep 18 19:15:00.786890 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:00.786852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:15:00.787109 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:00.787010    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:15:01.786436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:01.786399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:01.786920 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:01.786399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:01.786920 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:01.786522    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:15:01.786920 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:01.786635    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:15:02.786791 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:02.786751    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:15:02.787182 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:02.786869    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:15:03.422528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.422265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:03.422721 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.422431    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:03.422721 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.422676    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret podName:247b79f7-905d-4f9a-8dfc-078cea6f6aa7 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.422650087 +0000 UTC m=+67.798076376 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret") pod "global-pull-secret-syncer-mqnsc" (UID: "247b79f7-905d-4f9a-8dfc-078cea6f6aa7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:15:03.523535 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.523501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.523556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523657    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523683    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523699    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523711    2583 projected.go:196] Error preparing data for projected volume kube-api-access-jkfcs for pod openshift-network-diagnostics/network-check-target-xz56f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:03.523722 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523724    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.523704682 +0000 UTC m=+67.899130956 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:15:03.524018 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.523744    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs podName:4d05ddaa-3643-4909-8512-afa809925b25 nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.523733894 +0000 UTC m=+67.899160179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkfcs" (UniqueName: "kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs") pod "network-check-target-xz56f" (UID: "4d05ddaa-3643-4909-8512-afa809925b25") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:15:03.786586 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.786550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:03.786785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.786551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:03.786785 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.786690    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mqnsc" podUID="247b79f7-905d-4f9a-8dfc-078cea6f6aa7"
Sep 18 19:15:03.786785 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:03.786747    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xz56f" podUID="4d05ddaa-3643-4909-8512-afa809925b25"
Sep 18 19:15:03.926543 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.926513    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-205.ec2.internal" event="NodeReady"
Sep 18 19:15:03.926732 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:03.926657    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:15:04.067086 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.066996    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4g8h7"]
Sep 18 19:15:04.104514 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.104482    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xwl5f"]
Sep 18 19:15:04.105605 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.104933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.114872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.114813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:15:04.115065 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.115041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:15:04.115162 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.115064    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6m5tz\""
Sep 18 19:15:04.118252 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.118229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4g8h7"]
Sep 18 19:15:04.118375 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.118278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwl5f"]
Sep 18 19:15:04.118375 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.118353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.132326 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.132291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:15:04.132476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.132335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:15:04.132476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.132373    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmm6s\""
Sep 18 19:15:04.140826 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.140798    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:15:04.228332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-cert\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.228332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf4w\" (UniqueName: \"kubernetes.io/projected/22a1f40d-14f0-468d-83f8-d34b818e84ee-kube-api-access-kcf4w\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.228592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22a1f40d-14f0-468d-83f8-d34b818e84ee-config-volume\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.228592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsb8m\" (UniqueName: \"kubernetes.io/projected/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-kube-api-access-rsb8m\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.228592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a1f40d-14f0-468d-83f8-d34b818e84ee-tmp-dir\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.228592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.228539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.329870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22a1f40d-14f0-468d-83f8-d34b818e84ee-config-volume\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.329870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsb8m\" (UniqueName: \"kubernetes.io/projected/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-kube-api-access-rsb8m\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.329870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a1f40d-14f0-468d-83f8-d34b818e84ee-tmp-dir\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.330156 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.330156 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-cert\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.330156 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.329941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf4w\" (UniqueName: \"kubernetes.io/projected/22a1f40d-14f0-468d-83f8-d34b818e84ee-kube-api-access-kcf4w\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.330156 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:04.330046    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.330156 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:04.330138    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:04.830115368 +0000 UTC m=+37.205541642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.330521 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.330502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22a1f40d-14f0-468d-83f8-d34b818e84ee-config-volume\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.334872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.334844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-cert\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.340092 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.340063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a1f40d-14f0-468d-83f8-d34b818e84ee-tmp-dir\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.345261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.345237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf4w\" (UniqueName: \"kubernetes.io/projected/22a1f40d-14f0-468d-83f8-d34b818e84ee-kube-api-access-kcf4w\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.345474 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.345444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsb8m\" (UniqueName: \"kubernetes.io/projected/fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a-kube-api-access-rsb8m\") pod \"ingress-canary-xwl5f\" (UID: \"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a\") " pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.427131 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.427091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwl5f"
Sep 18 19:15:04.786417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.786379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:15:04.790861 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.790835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:04.791511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.790970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjcwp\""
Sep 18 19:15:04.833132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.833099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:04.833305 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:04.833251    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.833341 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:04.833327    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:05.833310308 +0000 UTC m=+38.208736579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:04.897270 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.897239    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwl5f"]
Sep 18 19:15:04.898572 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:15:04.898544    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcc952a_f8d1_4a6c_bcdb_a1e16dd66e2a.slice/crio-5239e2bf6623dc82616c3e5cec04fdb3d4ab1f43037af93ea0daa7152f8d8a57 WatchSource:0}: Error finding container 5239e2bf6623dc82616c3e5cec04fdb3d4ab1f43037af93ea0daa7152f8d8a57: Status 404 returned error can't find the container with id 5239e2bf6623dc82616c3e5cec04fdb3d4ab1f43037af93ea0daa7152f8d8a57
Sep 18 19:15:04.954722 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:04.954693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwl5f" event={"ID":"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a","Type":"ContainerStarted","Data":"5239e2bf6623dc82616c3e5cec04fdb3d4ab1f43037af93ea0daa7152f8d8a57"}
Sep 18 19:15:05.786877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.786658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:05.788070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.786691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:05.792244 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.792223    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:05.792895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.792875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:05.792895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.792890    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:05.793080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.792902    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-frfgm\""
Sep 18 19:15:05.842216 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.842184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:05.842413 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:05.842390    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:05.842474 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:05.842461    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:07.842442245 +0000 UTC m=+40.217868521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:05.961064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.961030    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="92e3623bce0baa1899c59db5032e8e9bbefd7f2133b4b4fd44d06c4e9f69d6dd" exitCode=0
Sep 18 19:15:05.961215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:05.961099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"92e3623bce0baa1899c59db5032e8e9bbefd7f2133b4b4fd44d06c4e9f69d6dd"}
Sep 18 19:15:06.967248 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:06.967212    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a905a6d-caab-41dd-8d3d-955f1ff08b0b" containerID="3b064ed4f7495707242c8a77fb9e03367439ecb95144d65ba41f429c47412990" exitCode=0
Sep 18 19:15:06.967721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:06.967276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerDied","Data":"3b064ed4f7495707242c8a77fb9e03367439ecb95144d65ba41f429c47412990"}
Sep 18 19:15:07.858750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:07.858694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:07.858930 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:07.858846    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:07.858930 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:07.858912    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:11.858894927 +0000 UTC m=+44.234321198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:07.971803 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:07.971768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwl5f" event={"ID":"fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a","Type":"ContainerStarted","Data":"14a5099ac28c94e4cd32e699a07c11adf9ce60cc367595de8a4aa9243a6e40cb"}
Sep 18 19:15:07.975080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:07.975051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" event={"ID":"6a905a6d-caab-41dd-8d3d-955f1ff08b0b","Type":"ContainerStarted","Data":"1bcba43ddbc8c37119289bb94b760cfe347955a2cd2c94582f05bb3fea22eda9"}
Sep 18 19:15:08.000410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:08.000332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xwl5f" podStartSLOduration=1.589500915 podStartE2EDuration="4.0003134s" totalImagesPullingTime="2.410812485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:07.999018098 +0000 UTC m=+40.374444390" watchObservedRunningTime="2026-09-18 19:15:08.0003134 +0000 UTC m=+40.375739690"
Sep 18 19:15:11.886330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:11.886288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:11.886803 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:11.886477    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:11.886803 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:11.886546    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:19.88652778 +0000 UTC m=+52.261954056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:19.941681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:19.941637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:19.942170 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:19.941821    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:19.942170 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:19.941911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:15:35.941886761 +0000 UTC m=+68.317313047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:28.952903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:28.952866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7fbdn"
Sep 18 19:15:28.994746 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:28.994689    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wz8cl" podStartSLOduration=30.204880479 podStartE2EDuration="58.994670745s" totalImagesPullingTime="28.789790266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:08.035884504 +0000 UTC m=+40.411310812" watchObservedRunningTime="2026-09-18 19:15:28.994670745 +0000 UTC m=+61.370097038"
Sep 18 19:15:35.451154 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.451108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:35.454707 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.454688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:15:35.464878 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.464843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/247b79f7-905d-4f9a-8dfc-078cea6f6aa7-original-pull-secret\") pod \"global-pull-secret-syncer-mqnsc\" (UID: \"247b79f7-905d-4f9a-8dfc-078cea6f6aa7\") " pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:35.498612 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.498577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mqnsc"
Sep 18 19:15:35.552460 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.552428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:35.552639 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.552501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:15:35.555834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.555809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:15:35.555971 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.555932    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:15:35.563118 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:35.563094    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:15:35.563192 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:35.563166    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:39.563143718 +0000 UTC m=+131.938569994 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : secret "metrics-daemon-secret" not found
Sep 18 19:15:35.566237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.566000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:15:35.576890 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.576856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfcs\" (UniqueName: \"kubernetes.io/projected/4d05ddaa-3643-4909-8512-afa809925b25-kube-api-access-jkfcs\") pod \"network-check-target-xz56f\" (UID: \"4d05ddaa-3643-4909-8512-afa809925b25\") " pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:35.804556 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.804523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mqnsc"]
Sep 18 19:15:35.807477 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.807451    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-frfgm\""
Sep 18 19:15:35.808027 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:15:35.808007    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247b79f7_905d_4f9a_8dfc_078cea6f6aa7.slice/crio-440df8d576e3302a64a9d5c6fd276fe37a2afa6b921e377e59a133c1d45400e7 WatchSource:0}: Error finding container 440df8d576e3302a64a9d5c6fd276fe37a2afa6b921e377e59a133c1d45400e7: Status 404 returned error can't find the container with id 440df8d576e3302a64a9d5c6fd276fe37a2afa6b921e377e59a133c1d45400e7
Sep 18 19:15:35.815510 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.815487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:35.955959 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:35.955913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:15:35.956143 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:35.956084    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:15:35.956186 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:15:35.956157    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:16:07.9561381 +0000 UTC m=+100.331564370 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:15:36.039881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:36.039838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqnsc" event={"ID":"247b79f7-905d-4f9a-8dfc-078cea6f6aa7","Type":"ContainerStarted","Data":"440df8d576e3302a64a9d5c6fd276fe37a2afa6b921e377e59a133c1d45400e7"}
Sep 18 19:15:36.115774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:36.115704    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xz56f"]
Sep 18 19:15:36.118971 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:15:36.118941    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d05ddaa_3643_4909_8512_afa809925b25.slice/crio-dc92abe54ee916495a62fd14f86ad37a20eed56b244efdfa8468fee1b193aed5 WatchSource:0}: Error finding container dc92abe54ee916495a62fd14f86ad37a20eed56b244efdfa8468fee1b193aed5: Status 404 returned error can't find the container with id dc92abe54ee916495a62fd14f86ad37a20eed56b244efdfa8468fee1b193aed5
Sep 18 19:15:37.045043 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:37.044938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xz56f" event={"ID":"4d05ddaa-3643-4909-8512-afa809925b25","Type":"ContainerStarted","Data":"dc92abe54ee916495a62fd14f86ad37a20eed56b244efdfa8468fee1b193aed5"}
Sep 18 19:15:41.057719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:41.057681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mqnsc" event={"ID":"247b79f7-905d-4f9a-8dfc-078cea6f6aa7","Type":"ContainerStarted","Data":"5a0458d3275d0f7a846002aa2f87b04133a1de112031730ae793a587c5e3db6d"}
Sep 18 19:15:41.059143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:41.059121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xz56f" event={"ID":"4d05ddaa-3643-4909-8512-afa809925b25","Type":"ContainerStarted","Data":"76efe043a112763801b5d51f16e59f368de2a696a28060fdc508850542bdcae7"}
Sep 18 19:15:41.059273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:41.059255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:15:41.077215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:41.077164    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mqnsc" podStartSLOduration=66.568798868 podStartE2EDuration="1m11.077145873s" totalImagesPullingTime="4.508347005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:41.076059 +0000 UTC m=+73.451485291" watchObservedRunningTime="2026-09-18 19:15:41.077145873 +0000 UTC m=+73.452572166"
Sep 18 19:15:41.096718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:15:41.096671    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xz56f" podStartSLOduration=67.902620291 podStartE2EDuration="1m12.096654862s" totalImagesPullingTime="4.194034571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:15:41.096479235 +0000 UTC m=+73.471905527" watchObservedRunningTime="2026-09-18 19:15:41.096654862 +0000 UTC m=+73.472081155"
Sep 18 19:16:07.965448 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:07.965407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:16:07.965909 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:07.965536    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:16:07.965909 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:07.965589    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls podName:22a1f40d-14f0-468d-83f8-d34b818e84ee nodeName:}" failed. No retries permitted until 2026-09-18 19:17:11.965575754 +0000 UTC m=+164.341002025 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls") pod "dns-default-4g8h7" (UID: "22a1f40d-14f0-468d-83f8-d34b818e84ee") : secret "dns-default-metrics-tls" not found
Sep 18 19:16:12.063530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:12.063499    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xz56f"
Sep 18 19:16:29.847000 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.846965    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"]
Sep 18 19:16:29.849528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.849510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:29.852389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.852343    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gv5nz\""
Sep 18 19:16:29.852670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.852650    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 19:16:29.853476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.853460    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:29.853579 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.853536    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:29.859709 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.859688    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-565c89cdb6-9s5v4"]
Sep 18 19:16:29.861398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.861383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.865414 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.865393    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 19:16:29.865518 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.865442    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 19:16:29.865585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.865569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vm76w\""
Sep 18 19:16:29.866191 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.866175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 19:16:29.866261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.866193    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 19:16:29.866261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.866195    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 19:16:29.870966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.870941    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"]
Sep 18 19:16:29.872546 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.872532    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 19:16:29.886794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.886762    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-565c89cdb6-9s5v4"]
Sep 18 19:16:29.912123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddll\" (UniqueName: \"kubernetes.io/projected/7ff79463-39de-48e7-afc6-0d5620677264-kube-api-access-pddll\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-default-certificate\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-stats-auth\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.912305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.912305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l95b\" (UniqueName: \"kubernetes.io/projected/bc5a596d-ba1c-4596-9f7a-d551506eaec6-kube-api-access-6l95b\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:29.966529 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.966495    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"]
Sep 18 19:16:29.968200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.968184    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-wbklz"]
Sep 18 19:16:29.968323 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.968303    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:29.969901 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.969878    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9kp6f"]
Sep 18 19:16:29.970034 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.970003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:29.971575 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.971550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:29.981641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.981618    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:29.981763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.981647    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 19:16:29.982556 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.982536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"]
Sep 18 19:16:29.983054 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:16:29.983147 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983058    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 19:16:29.983147 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983072    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:29.983263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983193    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:16:29.983771 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983718    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vqn89\""
Sep 18 19:16:29.983962 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983946    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:16:29.984027 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.983959    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:16:29.984713 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.984694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:29.984809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.984721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:16:29.985691 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.985669    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2js4g\""
Sep 18 19:16:29.986081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.985727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:16:29.986524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.986504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rlqcb\""
Sep 18 19:16:29.986775 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.986753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:16:29.988687 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.988663    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:16:29.998066 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.998047    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 19:16:29.998169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.998072    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 19:16:29.998418 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:29.998405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qtrf7\""
Sep 18 19:16:30.000186 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.000168    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 19:16:30.004524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.004500    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:16:30.005064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.005046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 19:16:30.007490 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.007463    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:30.008050 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.007933    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"]
Sep 18 19:16:30.008706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.008688    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"]
Sep 18 19:16:30.013605 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-tmp\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.013703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.013703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.013703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.013703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8831daaf-6e04-4e49-bc7d-9e5de158a32c-serving-cert\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.013764    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8ae4017-1968-4419-8996-91642ebd40ae-serving-cert\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrcv\" (UniqueName: \"kubernetes.io/projected/8831daaf-6e04-4e49-bc7d-9e5de158a32c-kube-api-access-gbrcv\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.013812    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:30.013846 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.013821    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:30.513802742 +0000 UTC m=+122.889229017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : secret "router-metrics-certs-default" not found
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.013857    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls podName:7ff79463-39de-48e7-afc6-0d5620677264 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:30.513844413 +0000 UTC m=+122.889270684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-krq84" (UID: "7ff79463-39de-48e7-afc6-0d5620677264") : secret "samples-operator-tls" not found
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8831daaf-6e04-4e49-bc7d-9e5de158a32c-config\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmfm\" (UniqueName: \"kubernetes.io/projected/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-kube-api-access-6nmfm\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l95b\" (UniqueName: \"kubernetes.io/projected/bc5a596d-ba1c-4596-9f7a-d551506eaec6-kube-api-access-6l95b\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.013970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjsxj\" (UniqueName: \"kubernetes.io/projected/c8ae4017-1968-4419-8996-91642ebd40ae-kube-api-access-qjsxj\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.014045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-config\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e130ed-b5d1-47e6-ad81-26d0cc582900-serving-cert\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-trusted-ca\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddll\" (UniqueName: \"kubernetes.io/projected/7ff79463-39de-48e7-afc6-0d5620677264-kube-api-access-pddll\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-snapshots\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-default-certificate\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.014310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.014523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-stats-auth\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.014523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.014351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xlj\" (UniqueName: \"kubernetes.io/projected/06e130ed-b5d1-47e6-ad81-26d0cc582900-kube-api-access-t5xlj\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.014523 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.014434    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:30.514421091 +0000 UTC m=+122.889847362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:30.016724 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.016700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-stats-auth\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.016892 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.016864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-default-certificate\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.023793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.023767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-wbklz"]
Sep 18 19:16:30.027136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.027113    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9kp6f"]
Sep 18 19:16:30.036109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.036088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l95b\" (UniqueName: \"kubernetes.io/projected/bc5a596d-ba1c-4596-9f7a-d551506eaec6-kube-api-access-6l95b\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.068067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.068028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddll\" (UniqueName: \"kubernetes.io/projected/7ff79463-39de-48e7-afc6-0d5620677264-kube-api-access-pddll\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:30.115238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-tmp\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.115238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.115238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.115238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8831daaf-6e04-4e49-bc7d-9e5de158a32c-serving-cert\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8ae4017-1968-4419-8996-91642ebd40ae-serving-cert\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrcv\" (UniqueName: \"kubernetes.io/projected/8831daaf-6e04-4e49-bc7d-9e5de158a32c-kube-api-access-gbrcv\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8831daaf-6e04-4e49-bc7d-9e5de158a32c-config\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmfm\" (UniqueName: \"kubernetes.io/projected/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-kube-api-access-6nmfm\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjsxj\" (UniqueName: \"kubernetes.io/projected/c8ae4017-1968-4419-8996-91642ebd40ae-kube-api-access-qjsxj\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-config\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e130ed-b5d1-47e6-ad81-26d0cc582900-serving-cert\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-trusted-ca\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.115606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-snapshots\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xlj\" (UniqueName: \"kubernetes.io/projected/06e130ed-b5d1-47e6-ad81-26d0cc582900-kube-api-access-t5xlj\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-tmp\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.115725    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.115814    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:16:30.615790511 +0000 UTC m=+122.991216806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.115906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.116132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.116068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.116466 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.116351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8831daaf-6e04-4e49-bc7d-9e5de158a32c-config\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.116714 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.116690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8ae4017-1968-4419-8996-91642ebd40ae-snapshots\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.116827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.116727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ae4017-1968-4419-8996-91642ebd40ae-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.116827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.116791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-config\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.117674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.117652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06e130ed-b5d1-47e6-ad81-26d0cc582900-trusted-ca\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.118204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.118185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8831daaf-6e04-4e49-bc7d-9e5de158a32c-serving-cert\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.118350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.118329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06e130ed-b5d1-47e6-ad81-26d0cc582900-serving-cert\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.118431 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.118418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8ae4017-1968-4419-8996-91642ebd40ae-serving-cert\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.161681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.161637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmfm\" (UniqueName: \"kubernetes.io/projected/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-kube-api-access-6nmfm\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.162186 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.162168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xlj\" (UniqueName: \"kubernetes.io/projected/06e130ed-b5d1-47e6-ad81-26d0cc582900-kube-api-access-t5xlj\") pod \"console-operator-fd4bb46c-wbklz\" (UID: \"06e130ed-b5d1-47e6-ad81-26d0cc582900\") " pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.165514 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.165488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrcv\" (UniqueName: \"kubernetes.io/projected/8831daaf-6e04-4e49-bc7d-9e5de158a32c-kube-api-access-gbrcv\") pod \"service-ca-operator-57cdc4b768-8tm25\" (UID: \"8831daaf-6e04-4e49-bc7d-9e5de158a32c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.166343 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.166314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjsxj\" (UniqueName: \"kubernetes.io/projected/c8ae4017-1968-4419-8996-91642ebd40ae-kube-api-access-qjsxj\") pod \"insights-operator-6f9c644dd9-9kp6f\" (UID: \"c8ae4017-1968-4419-8996-91642ebd40ae\") " pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.204843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.204809    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:16:30.207447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.207431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.221657 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.221629    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:16:30.222040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.222025    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:16:30.222259 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.222245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:16:30.222433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.222420    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jqgxp\""
Sep 18 19:16:30.242985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.242963    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:16:30.284792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.284753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:30.292803 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.292784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f"
Sep 18 19:16:30.297737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.297714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"
Sep 18 19:16:30.317171 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317339 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317499 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317581 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.317627 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.317613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxtl9\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.335309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.335189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:16:30.418583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxtl9\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418645    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.418789    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.418808    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54c64c4d7-hscg4: secret "image-registry-tls" not found
Sep 18 19:16:30.418859 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.418865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls podName:9540b991-d8d3-4381-a4e7-576181a4cb79 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:30.918845332 +0000 UTC m=+123.294271624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls") pod "image-registry-54c64c4d7-hscg4" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79") : secret "image-registry-tls" not found
Sep 18 19:16:30.419148 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.418907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.419210 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.419189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.419309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.419284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.419780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.419757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.421166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.421142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.421261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.421146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.446384 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.446345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxtl9\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.446935 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.446918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.519661 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.519622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:30.519821 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.519771    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:30.519821 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.519788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.519905 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.519822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:30.519905 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.519846    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls podName:7ff79463-39de-48e7-afc6-0d5620677264 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.519829581 +0000 UTC m=+123.895255852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-krq84" (UID: "7ff79463-39de-48e7-afc6-0d5620677264") : secret "samples-operator-tls" not found
Sep 18 19:16:30.519905 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.519902    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:16:30.520027 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.519945    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.519934671 +0000 UTC m=+123.895360943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : secret "router-metrics-certs-default" not found
Sep 18 19:16:30.520027 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.519958    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.519952174 +0000 UTC m=+123.895378473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:30.567880 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.567843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-wbklz"]
Sep 18 19:16:30.570304 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:16:30.570270    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e130ed_b5d1_47e6_ad81_26d0cc582900.slice/crio-c19684956909769f55d1bdb8aa6b39c62b45e44d070a07ef0192ec64ca31bf1b WatchSource:0}: Error finding container c19684956909769f55d1bdb8aa6b39c62b45e44d070a07ef0192ec64ca31bf1b: Status 404 returned error can't find the container with id c19684956909769f55d1bdb8aa6b39c62b45e44d070a07ef0192ec64ca31bf1b
Sep 18 19:16:30.597536 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.597501    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-9kp6f"]
Sep 18 19:16:30.600182 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.600159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25"]
Sep 18 19:16:30.602677 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:16:30.602652    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae4017_1968_4419_8996_91642ebd40ae.slice/crio-d1713a84e8a2a1227c8317743af72c84094a2b00c34b97fdc9fc14a853e9d272 WatchSource:0}: Error finding container d1713a84e8a2a1227c8317743af72c84094a2b00c34b97fdc9fc14a853e9d272: Status 404 returned error can't find the container with id d1713a84e8a2a1227c8317743af72c84094a2b00c34b97fdc9fc14a853e9d272
Sep 18 19:16:30.603163 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:16:30.603126    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8831daaf_6e04_4e49_bc7d_9e5de158a32c.slice/crio-c774862771af69d3fb63eba90e9b823acd67e032276a33431a76854eaeb4ffd8 WatchSource:0}: Error finding container c774862771af69d3fb63eba90e9b823acd67e032276a33431a76854eaeb4ffd8: Status 404 returned error can't find the container with id c774862771af69d3fb63eba90e9b823acd67e032276a33431a76854eaeb4ffd8
Sep 18 19:16:30.621070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.621030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:30.621226 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.621202    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:30.621312 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.621284    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.621263819 +0000 UTC m=+123.996690089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:30.924644 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:30.924603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:30.925058 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.924729    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:30.925058 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.924742    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54c64c4d7-hscg4: secret "image-registry-tls" not found
Sep 18 19:16:30.925058 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:30.924789    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls podName:9540b991-d8d3-4381-a4e7-576181a4cb79 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:31.924776639 +0000 UTC m=+124.300202910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls") pod "image-registry-54c64c4d7-hscg4" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79") : secret "image-registry-tls" not found
Sep 18 19:16:31.187589 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.187500    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25" event={"ID":"8831daaf-6e04-4e49-bc7d-9e5de158a32c","Type":"ContainerStarted","Data":"c774862771af69d3fb63eba90e9b823acd67e032276a33431a76854eaeb4ffd8"}
Sep 18 19:16:31.189016 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.188945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f" event={"ID":"c8ae4017-1968-4419-8996-91642ebd40ae","Type":"ContainerStarted","Data":"d1713a84e8a2a1227c8317743af72c84094a2b00c34b97fdc9fc14a853e9d272"}
Sep 18 19:16:31.190397 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.190330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" event={"ID":"06e130ed-b5d1-47e6-ad81-26d0cc582900","Type":"ContainerStarted","Data":"c19684956909769f55d1bdb8aa6b39c62b45e44d070a07ef0192ec64ca31bf1b"}
Sep 18 19:16:31.531937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.531886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:31.532133 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.531959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:31.532133 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.531991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:31.532133 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.532053    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.532034289 +0000 UTC m=+125.907460572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:31.532133 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.532117    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:16:31.532347 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.532141    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:31.532347 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.532182    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.532163649 +0000 UTC m=+125.907589937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : secret "router-metrics-certs-default" not found
Sep 18 19:16:31.532347 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.532205    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls podName:7ff79463-39de-48e7-afc6-0d5620677264 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.532191698 +0000 UTC m=+125.907617972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-krq84" (UID: "7ff79463-39de-48e7-afc6-0d5620677264") : secret "samples-operator-tls" not found
Sep 18 19:16:31.632739 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.632693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:31.632979 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.632963    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:31.633053 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.633030    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.6330122 +0000 UTC m=+126.008438475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:31.935681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:31.935580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:31.936144 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.935723    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:31.936144 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.935747    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54c64c4d7-hscg4: secret "image-registry-tls" not found
Sep 18 19:16:31.936144 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:31.935814    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls podName:9540b991-d8d3-4381-a4e7-576181a4cb79 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:33.935793433 +0000 UTC m=+126.311219705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls") pod "image-registry-54c64c4d7-hscg4" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79") : secret "image-registry-tls" not found
Sep 18 19:16:33.550304 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:33.550266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:33.550304 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:33.550307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.550440    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.550460    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.550498    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls podName:7ff79463-39de-48e7-afc6-0d5620677264 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:37.550479324 +0000 UTC m=+129.925905597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-krq84" (UID: "7ff79463-39de-48e7-afc6-0d5620677264") : secret "samples-operator-tls" not found
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.550524    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:37.550505514 +0000 UTC m=+129.925931801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : secret "router-metrics-certs-default" not found
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:33.550631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:33.550796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.550752    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:37.550739859 +0000 UTC m=+129.926166132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:33.651189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:33.651142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:33.651408 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.651286    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:33.651408 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.651382    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:16:37.651348861 +0000 UTC m=+130.026775147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:33.954672 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:33.954636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:33.954825 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.954759    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:33.954825 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.954772    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54c64c4d7-hscg4: secret "image-registry-tls" not found
Sep 18 19:16:33.954825 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:33.954820    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls podName:9540b991-d8d3-4381-a4e7-576181a4cb79 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:37.954806182 +0000 UTC m=+130.330232454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls") pod "image-registry-54c64c4d7-hscg4" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79") : secret "image-registry-tls" not found
Sep 18 19:16:34.204771 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.204673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f" event={"ID":"c8ae4017-1968-4419-8996-91642ebd40ae","Type":"ContainerStarted","Data":"bc1ecb4de8f1896219c8bfba6a3675ad414f079952561b26bc1fe72961589a00"}
Sep 18 19:16:34.206530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.206508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/0.log"
Sep 18 19:16:34.206670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.206543    2583 generic.go:356] "Generic (PLEG): container finished" podID="06e130ed-b5d1-47e6-ad81-26d0cc582900" containerID="ba5a1c266ba599e862dd7a339ab3d7b4faa55af24017390e0c2d7cce15dd2e8e" exitCode=255
Sep 18 19:16:34.206670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.206644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" event={"ID":"06e130ed-b5d1-47e6-ad81-26d0cc582900","Type":"ContainerDied","Data":"ba5a1c266ba599e862dd7a339ab3d7b4faa55af24017390e0c2d7cce15dd2e8e"}
Sep 18 19:16:34.206842 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.206824    2583 scope.go:117] "RemoveContainer" containerID="ba5a1c266ba599e862dd7a339ab3d7b4faa55af24017390e0c2d7cce15dd2e8e"
Sep 18 19:16:34.208372 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.208331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25" event={"ID":"8831daaf-6e04-4e49-bc7d-9e5de158a32c","Type":"ContainerStarted","Data":"471eb48fa22f865716b8e52a4cff2e23da3340f7c406a10e612aa998d73d45ce"}
Sep 18 19:16:34.249153 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.249092    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f" podStartSLOduration=2.388850366 podStartE2EDuration="5.249071364s" totalImagesPullingTime="2.860220998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:34.248303451 +0000 UTC m=+126.623729767" watchObservedRunningTime="2026-09-18 19:16:34.249071364 +0000 UTC m=+126.624497655"
Sep 18 19:16:34.251627 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.251598    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"]
Sep 18 19:16:34.254639 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.254619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"
Sep 18 19:16:34.274844 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.274804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wdtcb\""
Sep 18 19:16:34.274999 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.274848    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 19:16:34.276957 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.276938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 19:16:34.295934 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.295897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"]
Sep 18 19:16:34.312497 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.312433    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25" podStartSLOduration=2.449768685 podStartE2EDuration="5.312410651s" totalImagesPullingTime="2.862641966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:34.309922879 +0000 UTC m=+126.685349173" watchObservedRunningTime="2026-09-18 19:16:34.312410651 +0000 UTC m=+126.687836944"
Sep 18 19:16:34.357994 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.357962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsv9\" (UniqueName: \"kubernetes.io/projected/de32a182-70c4-4c37-9f9b-35c766d1ff4f-kube-api-access-qgsv9\") pod \"migrator-76794cbd6c-ff4fx\" (UID: \"de32a182-70c4-4c37-9f9b-35c766d1ff4f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"
Sep 18 19:16:34.458861 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.458768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsv9\" (UniqueName: \"kubernetes.io/projected/de32a182-70c4-4c37-9f9b-35c766d1ff4f-kube-api-access-qgsv9\") pod \"migrator-76794cbd6c-ff4fx\" (UID: \"de32a182-70c4-4c37-9f9b-35c766d1ff4f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"
Sep 18 19:16:34.470229 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.470199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsv9\" (UniqueName: \"kubernetes.io/projected/de32a182-70c4-4c37-9f9b-35c766d1ff4f-kube-api-access-qgsv9\") pod \"migrator-76794cbd6c-ff4fx\" (UID: \"de32a182-70c4-4c37-9f9b-35c766d1ff4f\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"
Sep 18 19:16:34.566868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.566821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"
Sep 18 19:16:34.858740 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.858684    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx"]
Sep 18 19:16:34.863592 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:16:34.863563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde32a182_70c4_4c37_9f9b_35c766d1ff4f.slice/crio-5fe5c684c0ff67f191bdce5e9a68c5daca5f12d5fab34af6ae2ee8c94a6421b4 WatchSource:0}: Error finding container 5fe5c684c0ff67f191bdce5e9a68c5daca5f12d5fab34af6ae2ee8c94a6421b4: Status 404 returned error can't find the container with id 5fe5c684c0ff67f191bdce5e9a68c5daca5f12d5fab34af6ae2ee8c94a6421b4
Sep 18 19:16:34.938902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.938868    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gzmkc"]
Sep 18 19:16:34.941504 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.941485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:34.944274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.944256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xd5sl\""
Sep 18 19:16:34.944583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.944568    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:16:34.944655 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.944610    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:16:34.953932 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:34.953909    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gzmkc"]
Sep 18 19:16:35.064112 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.064075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.064288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.064129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbxs\" (UniqueName: \"kubernetes.io/projected/1215df9a-994e-45a1-86ec-9c48436b5718-kube-api-access-4rbxs\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.064288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.064159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.064288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.064280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1215df9a-994e-45a1-86ec-9c48436b5718-data-volume\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.064421 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.064318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1215df9a-994e-45a1-86ec-9c48436b5718-crio-socket\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.165546 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.165546 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbxs\" (UniqueName: \"kubernetes.io/projected/1215df9a-994e-45a1-86ec-9c48436b5718-kube-api-access-4rbxs\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.165611    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.165674    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.66565746 +0000 UTC m=+128.041083757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.165701    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.165754    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:35.665737954 +0000 UTC m=+128.041164237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.165815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1215df9a-994e-45a1-86ec-9c48436b5718-data-volume\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.166004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1215df9a-994e-45a1-86ec-9c48436b5718-crio-socket\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.166004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.165926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1215df9a-994e-45a1-86ec-9c48436b5718-crio-socket\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.166099 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.166083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1215df9a-994e-45a1-86ec-9c48436b5718-data-volume\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.176780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.176746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbxs\" (UniqueName: \"kubernetes.io/projected/1215df9a-994e-45a1-86ec-9c48436b5718-kube-api-access-4rbxs\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.213476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.213431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx" event={"ID":"de32a182-70c4-4c37-9f9b-35c766d1ff4f","Type":"ContainerStarted","Data":"5fe5c684c0ff67f191bdce5e9a68c5daca5f12d5fab34af6ae2ee8c94a6421b4"}
Sep 18 19:16:35.214849 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.214831    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/1.log"
Sep 18 19:16:35.215193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.215179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/0.log"
Sep 18 19:16:35.215241 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.215211    2583 generic.go:356] "Generic (PLEG): container finished" podID="06e130ed-b5d1-47e6-ad81-26d0cc582900" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654" exitCode=255
Sep 18 19:16:35.215280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.215240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" event={"ID":"06e130ed-b5d1-47e6-ad81-26d0cc582900","Type":"ContainerDied","Data":"e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"}
Sep 18 19:16:35.215324 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.215310    2583 scope.go:117] "RemoveContainer" containerID="ba5a1c266ba599e862dd7a339ab3d7b4faa55af24017390e0c2d7cce15dd2e8e"
Sep 18 19:16:35.215612 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.215589    2583 scope.go:117] "RemoveContainer" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"
Sep 18 19:16:35.215824 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.215803    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-wbklz_openshift-console-operator(06e130ed-b5d1-47e6-ad81-26d0cc582900)\"" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podUID="06e130ed-b5d1-47e6-ad81-26d0cc582900"
Sep 18 19:16:35.669931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.669877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.670426 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.670023    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.670426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:35.670078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:35.670426 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.670098    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:36.670077389 +0000 UTC m=+129.045503675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:35.670426 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.670203    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:35.670426 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:35.670270    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:36.670252889 +0000 UTC m=+129.045679178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:36.220683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.220643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx" event={"ID":"de32a182-70c4-4c37-9f9b-35c766d1ff4f","Type":"ContainerStarted","Data":"b3bafe38f08186982ed0ce5163cfd02111639b43a4a303103c069159a7495c65"}
Sep 18 19:16:36.220878 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.220717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx" event={"ID":"de32a182-70c4-4c37-9f9b-35c766d1ff4f","Type":"ContainerStarted","Data":"d12a805fd2a4bd6926c3a364d940f052e7f3d177d63acc91135e86a41ee766cc"}
Sep 18 19:16:36.222455 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.222434    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/1.log"
Sep 18 19:16:36.222745 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.222730    2583 scope.go:117] "RemoveContainer" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"
Sep 18 19:16:36.222893 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:36.222877    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-wbklz_openshift-console-operator(06e130ed-b5d1-47e6-ad81-26d0cc582900)\"" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podUID="06e130ed-b5d1-47e6-ad81-26d0cc582900"
Sep 18 19:16:36.237485 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.237423    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-ff4fx" podStartSLOduration=1.336753956 podStartE2EDuration="2.237404933s" totalImagesPullingTime="900.650977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:36.236902715 +0000 UTC m=+128.612329009" watchObservedRunningTime="2026-09-18 19:16:36.237404933 +0000 UTC m=+128.612831224"
Sep 18 19:16:36.678686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.678632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:36.679082 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:36.678704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:36.679082 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:36.678782    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:36.679082 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:36.678788    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:36.679082 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:36.678833    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:38.678820126 +0000 UTC m=+131.054246398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:36.679082 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:36.678846    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:38.67883993 +0000 UTC m=+131.054266201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:37.411341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.411315    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4zxvd_561e9854-ab29-47f0-b3e0-620407e4d964/dns-node-resolver/0.log"
Sep 18 19:16:37.586628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.586585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:37.586850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.586656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:37.586850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.586690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:37.586850 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.586765    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:45.586742979 +0000 UTC m=+137.962169250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:37.586850 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.586826    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 19:16:37.587037 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.586879    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:45.586864823 +0000 UTC m=+137.962291113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : secret "router-metrics-certs-default" not found
Sep 18 19:16:37.587037 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.586826    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 19:16:37.587037 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.586920    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls podName:7ff79463-39de-48e7-afc6-0d5620677264 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:45.58691005 +0000 UTC m=+137.962336322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-krq84" (UID: "7ff79463-39de-48e7-afc6-0d5620677264") : secret "samples-operator-tls" not found
Sep 18 19:16:37.687613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.687517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:37.688063 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.687681    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:37.688063 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.687775    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:16:45.687752487 +0000 UTC m=+138.063178760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:37.990654 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:37.990547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:37.990807 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.990696    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:16:37.990807 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.990719    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54c64c4d7-hscg4: secret "image-registry-tls" not found
Sep 18 19:16:37.990807 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:37.990772    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls podName:9540b991-d8d3-4381-a4e7-576181a4cb79 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:45.990757507 +0000 UTC m=+138.366183778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls") pod "image-registry-54c64c4d7-hscg4" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79") : secret "image-registry-tls" not found
Sep 18 19:16:38.696295 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:38.696235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:38.696700 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:38.696405    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:38.696700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:38.696438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:38.696700 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:38.696470    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:42.696454528 +0000 UTC m=+135.071880804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:38.696700 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:38.696494    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:38.696700 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:38.696549    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:42.696535449 +0000 UTC m=+135.071961729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:38.809444 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:38.809411    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xh8cv_faf1d939-4798-4480-bb3d-ba0f6cd73c6a/node-ca/0.log"
Sep 18 19:16:39.603867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:39.603834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwl5f_fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a/serve-healthcheck-canary/0.log"
Sep 18 19:16:39.604046 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:39.604028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:16:39.604201 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:39.604184    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:16:39.604253 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:39.604243    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs podName:9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04 nodeName:}" failed. No retries permitted until 2026-09-18 19:18:41.604229086 +0000 UTC m=+253.979655362 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs") pod "network-metrics-daemon-nsgdm" (UID: "9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04") : secret "metrics-daemon-secret" not found
Sep 18 19:16:39.809645 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:39.809614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-ff4fx_de32a182-70c4-4c37-9f9b-35c766d1ff4f/migrator/0.log"
Sep 18 19:16:40.017076 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:40.017045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-ff4fx_de32a182-70c4-4c37-9f9b-35c766d1ff4f/graceful-termination/0.log"
Sep 18 19:16:40.285506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:40.285421    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:40.285506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:40.285478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:16:40.285831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:40.285819    2583 scope.go:117] "RemoveContainer" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"
Sep 18 19:16:40.285992 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:40.285974    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-wbklz_openshift-console-operator(06e130ed-b5d1-47e6-ad81-26d0cc582900)\"" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podUID="06e130ed-b5d1-47e6-ad81-26d0cc582900"
Sep 18 19:16:42.732152 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:42.732117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:42.732549 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:42.732229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:42.732549 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:42.732242    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:16:42.732549 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:42.732304    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:50.732288433 +0000 UTC m=+143.107714717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : configmap "kube-rbac-proxy" not found
Sep 18 19:16:42.732549 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:42.732318    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:42.732549 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:42.732356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:16:50.732345443 +0000 UTC m=+143.107771713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:45.656018 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.655977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:45.656398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.656157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:45.656398 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.656186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:45.656398 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:45.656338    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle podName:bc5a596d-ba1c-4596-9f7a-d551506eaec6 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:01.65631283 +0000 UTC m=+154.031739121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle") pod "router-default-565c89cdb6-9s5v4" (UID: "bc5a596d-ba1c-4596-9f7a-d551506eaec6") : configmap references non-existent config key: service-ca.crt
Sep 18 19:16:45.658440 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.658419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc5a596d-ba1c-4596-9f7a-d551506eaec6-metrics-certs\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:16:45.658551 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.658473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ff79463-39de-48e7-afc6-0d5620677264-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-krq84\" (UID: \"7ff79463-39de-48e7-afc6-0d5620677264\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:45.756791 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.756753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:16:45.756965 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:45.756906    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:45.757013 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:45.756975    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls podName:7fb29efd-aee5-4e8e-918e-96d40d8c75bc nodeName:}" failed. No retries permitted until 2026-09-18 19:17:01.756959177 +0000 UTC m=+154.132385454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-mnhtw" (UID: "7fb29efd-aee5-4e8e-918e-96d40d8c75bc") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:16:45.757573 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:45.757552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"
Sep 18 19:16:46.053293 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.053262    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84"]
Sep 18 19:16:46.059440 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.059411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:46.061730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.061703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"image-registry-54c64c4d7-hscg4\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") " pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:46.115946 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.115923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:46.252678 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.252641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84" event={"ID":"7ff79463-39de-48e7-afc6-0d5620677264","Type":"ContainerStarted","Data":"4d9786073fb17719a61f744c9b6298ef22e66af66030b912a6edfb24bd53caac"}
Sep 18 19:16:46.402437 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:46.402341    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:16:46.405499 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:16:46.405473    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9540b991_d8d3_4381_a4e7_576181a4cb79.slice/crio-fca84956ef2bdb70ca9c143f9bc7680e521c1b9522d24fc15a55f7099a5d31a9 WatchSource:0}: Error finding container fca84956ef2bdb70ca9c143f9bc7680e521c1b9522d24fc15a55f7099a5d31a9: Status 404 returned error can't find the container with id fca84956ef2bdb70ca9c143f9bc7680e521c1b9522d24fc15a55f7099a5d31a9
Sep 18 19:16:47.257894 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:47.257857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" event={"ID":"9540b991-d8d3-4381-a4e7-576181a4cb79","Type":"ContainerStarted","Data":"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"}
Sep 18 19:16:47.258245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:47.257918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" event={"ID":"9540b991-d8d3-4381-a4e7-576181a4cb79","Type":"ContainerStarted","Data":"fca84956ef2bdb70ca9c143f9bc7680e521c1b9522d24fc15a55f7099a5d31a9"}
Sep 18 19:16:47.258245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:47.257959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:16:48.263092 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:48.263049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84" event={"ID":"7ff79463-39de-48e7-afc6-0d5620677264","Type":"ContainerStarted","Data":"6d6432a2c1ac6396c43865538e4083b0f2928ca4ac96397abbcf0af1c7ce58b8"}
Sep 18 19:16:48.263502 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:48.263109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84" event={"ID":"7ff79463-39de-48e7-afc6-0d5620677264","Type":"ContainerStarted","Data":"3b1b2094e4dc0ef393582a04b95993b14448f44e2e59be1af15a367e5ac9b967"}
Sep 18 19:16:48.333614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:48.333560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-krq84" podStartSLOduration=17.718085392 podStartE2EDuration="19.333543799s" totalImagesPullingTime="1.615458407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:48.33326962 +0000 UTC m=+140.708695912" watchObservedRunningTime="2026-09-18 19:16:48.333543799 +0000 UTC m=+140.708970091"
Sep 18 19:16:48.333792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:48.333716    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" podStartSLOduration=18.333707808 podStartE2EDuration="18.333707808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:16:47.305480071 +0000 UTC m=+139.680906364" watchObservedRunningTime="2026-09-18 19:16:48.333707808 +0000 UTC m=+140.709134101"
Sep 18 19:16:50.787297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:50.787264    2583 scope.go:117] "RemoveContainer" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"
Sep 18 19:16:50.803586 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:50.803563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:50.803675 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:50.803606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:50.803741 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:50.803725    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:50.803796 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:50.803786    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls podName:1215df9a-994e-45a1-86ec-9c48436b5718 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:06.803768393 +0000 UTC m=+159.179194677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gzmkc" (UID: "1215df9a-994e-45a1-86ec-9c48436b5718") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:16:50.804158 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:50.804136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1215df9a-994e-45a1-86ec-9c48436b5718-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:16:51.273750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.273723    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:16:51.274127 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.274113    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/1.log"
Sep 18 19:16:51.274187 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.274144    2583 generic.go:356] "Generic (PLEG): container finished" podID="06e130ed-b5d1-47e6-ad81-26d0cc582900" containerID="ba8e3b6223211179e17f0e81c4798021cfb0bb8d345bcc6e6f00b343a9319efc" exitCode=255
Sep 18 19:16:51.274187 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.274176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" event={"ID":"06e130ed-b5d1-47e6-ad81-26d0cc582900","Type":"ContainerDied","Data":"ba8e3b6223211179e17f0e81c4798021cfb0bb8d345bcc6e6f00b343a9319efc"}
Sep 18 19:16:51.274261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.274216    2583 scope.go:117] "RemoveContainer" containerID="e43089b698348ca9af5d034815f7f1c04e61b222bcc2950444ec357da19b0654"
Sep 18 19:16:51.274621 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:51.274601    2583 scope.go:117] "RemoveContainer" containerID="ba8e3b6223211179e17f0e81c4798021cfb0bb8d345bcc6e6f00b343a9319efc"
Sep 18 19:16:51.274801 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:16:51.274776    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-wbklz_openshift-console-operator(06e130ed-b5d1-47e6-ad81-26d0cc582900)\"" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podUID="06e130ed-b5d1-47e6-ad81-26d0cc582900"
Sep 18 19:16:52.279260 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:16:52.279233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:17:00.285012 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:00.284961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:17:00.285486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:00.285062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:17:00.285486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:00.285390    2583 scope.go:117] "RemoveContainer" containerID="ba8e3b6223211179e17f0e81c4798021cfb0bb8d345bcc6e6f00b343a9319efc"
Sep 18 19:17:00.285586 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:00.285550    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-wbklz_openshift-console-operator(06e130ed-b5d1-47e6-ad81-26d0cc582900)\"" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podUID="06e130ed-b5d1-47e6-ad81-26d0cc582900"
Sep 18 19:17:01.691358 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.691313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:01.691960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.691938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc5a596d-ba1c-4596-9f7a-d551506eaec6-service-ca-bundle\") pod \"router-default-565c89cdb6-9s5v4\" (UID: \"bc5a596d-ba1c-4596-9f7a-d551506eaec6\") " pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:01.791890 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.791854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:17:01.794248 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.794223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fb29efd-aee5-4e8e-918e-96d40d8c75bc-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-mnhtw\" (UID: \"7fb29efd-aee5-4e8e-918e-96d40d8c75bc\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:17:01.969850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.969759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:01.999143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:01.999111    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:17:02.003958 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:02.003911    2583 patch_prober.go:28] interesting pod/image-registry-54c64c4d7-hscg4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 19:17:02.004155 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:02.003997    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 19:17:02.078498 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:02.078457    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"
Sep 18 19:17:02.309943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:02.309912    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-565c89cdb6-9s5v4"]
Sep 18 19:17:02.313007 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:02.312977    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5a596d_ba1c_4596_9f7a_d551506eaec6.slice/crio-2c389e19891746d7652923fbb0a077a322152b6f62b0c29d628fb6bd01de428c WatchSource:0}: Error finding container 2c389e19891746d7652923fbb0a077a322152b6f62b0c29d628fb6bd01de428c: Status 404 returned error can't find the container with id 2c389e19891746d7652923fbb0a077a322152b6f62b0c29d628fb6bd01de428c
Sep 18 19:17:02.395920 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:02.395888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb29efd_aee5_4e8e_918e_96d40d8c75bc.slice/crio-d7dc70898a48962a3817b7f0b373a61c1fdb0453fd6a7adda8f9e0521c3eec62 WatchSource:0}: Error finding container d7dc70898a48962a3817b7f0b373a61c1fdb0453fd6a7adda8f9e0521c3eec62: Status 404 returned error can't find the container with id d7dc70898a48962a3817b7f0b373a61c1fdb0453fd6a7adda8f9e0521c3eec62
Sep 18 19:17:02.396937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:02.396853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw"]
Sep 18 19:17:03.318380 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.318327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-565c89cdb6-9s5v4" event={"ID":"bc5a596d-ba1c-4596-9f7a-d551506eaec6","Type":"ContainerStarted","Data":"2555e890abcabdaea98d131005b571e9a1782ecf8947a42147e95caf56c3d1d6"}
Sep 18 19:17:03.318864 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.318410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-565c89cdb6-9s5v4" event={"ID":"bc5a596d-ba1c-4596-9f7a-d551506eaec6","Type":"ContainerStarted","Data":"2c389e19891746d7652923fbb0a077a322152b6f62b0c29d628fb6bd01de428c"}
Sep 18 19:17:03.319954 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.319925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw" event={"ID":"7fb29efd-aee5-4e8e-918e-96d40d8c75bc","Type":"ContainerStarted","Data":"d7dc70898a48962a3817b7f0b373a61c1fdb0453fd6a7adda8f9e0521c3eec62"}
Sep 18 19:17:03.352186 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.352134    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-565c89cdb6-9s5v4" podStartSLOduration=34.352114972 podStartE2EDuration="34.352114972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:03.350898522 +0000 UTC m=+155.726324815" watchObservedRunningTime="2026-09-18 19:17:03.352114972 +0000 UTC m=+155.727541264"
Sep 18 19:17:03.970389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.970336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:03.973349 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:03.973319    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:04.326092 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:04.326045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw" event={"ID":"7fb29efd-aee5-4e8e-918e-96d40d8c75bc","Type":"ContainerStarted","Data":"4ed5b1f97894f3ee2b7d7b6254a617d90ca62251717e82bdeb43c38405fb4abd"}
Sep 18 19:17:04.326568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:04.326211    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:04.327648 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:04.327628    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-565c89cdb6-9s5v4"
Sep 18 19:17:04.348784 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:04.348727    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-mnhtw" podStartSLOduration=33.577680913 podStartE2EDuration="35.348707466s" totalImagesPullingTime="1.771026553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:04.348203176 +0000 UTC m=+156.723629469" watchObservedRunningTime="2026-09-18 19:17:04.348707466 +0000 UTC m=+156.724133840"
Sep 18 19:17:06.832241 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:06.832189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:17:06.834503 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:06.834481    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1215df9a-994e-45a1-86ec-9c48436b5718-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gzmkc\" (UID: \"1215df9a-994e-45a1-86ec-9c48436b5718\") " pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:17:07.051141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:07.051104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gzmkc"
Sep 18 19:17:07.119877 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:07.119833    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4g8h7" podUID="22a1f40d-14f0-468d-83f8-d34b818e84ee"
Sep 18 19:17:07.336550 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:07.336517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:07.343910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:07.343891    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gzmkc"]
Sep 18 19:17:07.346179 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:07.346155    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1215df9a_994e_45a1_86ec_9c48436b5718.slice/crio-17613e02269e2d5654132a311321e33775813b04b510afc582f7c1082b8051b6 WatchSource:0}: Error finding container 17613e02269e2d5654132a311321e33775813b04b510afc582f7c1082b8051b6: Status 404 returned error can't find the container with id 17613e02269e2d5654132a311321e33775813b04b510afc582f7c1082b8051b6
Sep 18 19:17:07.795597 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:07.795558    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nsgdm" podUID="9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04"
Sep 18 19:17:08.341452 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.341421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzmkc" event={"ID":"1215df9a-994e-45a1-86ec-9c48436b5718","Type":"ContainerStarted","Data":"d77aff38941bc118901d2db44d50cd5eb3980d13e610006ddf8b47111470b00a"}
Sep 18 19:17:08.341769 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.341479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzmkc" event={"ID":"1215df9a-994e-45a1-86ec-9c48436b5718","Type":"ContainerStarted","Data":"17613e02269e2d5654132a311321e33775813b04b510afc582f7c1082b8051b6"}
Sep 18 19:17:08.976850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.976814    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"]
Sep 18 19:17:08.988725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.988694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:08.994335 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.994304    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 19:17:08.994335 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.994331    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 19:17:08.996051 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.995679    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rmm92\""
Sep 18 19:17:08.996051 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.995755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:17:08.997694 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:08.997669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"]
Sep 18 19:17:09.051224 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.051186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8fzn\" (UniqueName: \"kubernetes.io/projected/c4283428-426d-4c47-aa9f-8533caee3298-kube-api-access-s8fzn\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.051436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.051231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4283428-426d-4c47-aa9f-8533caee3298-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.051436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.051292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.051436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.051311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.152502 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.152461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8fzn\" (UniqueName: \"kubernetes.io/projected/c4283428-426d-4c47-aa9f-8533caee3298-kube-api-access-s8fzn\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.152699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.152525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4283428-426d-4c47-aa9f-8533caee3298-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.152699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.152617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.152699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.152678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.164676 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.164645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4283428-426d-4c47-aa9f-8533caee3298-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.171151 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.171112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.171330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.171231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4283428-426d-4c47-aa9f-8533caee3298-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.171330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.171248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8fzn\" (UniqueName: \"kubernetes.io/projected/c4283428-426d-4c47-aa9f-8533caee3298-kube-api-access-s8fzn\") pod \"prometheus-operator-7bd5d56d6-xttd2\" (UID: \"c4283428-426d-4c47-aa9f-8533caee3298\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.301142 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.301104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"
Sep 18 19:17:09.353822 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.353782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzmkc" event={"ID":"1215df9a-994e-45a1-86ec-9c48436b5718","Type":"ContainerStarted","Data":"6590aec70091bdeef3aaf1acb74e7ef76556ec718e89ecb5f1420ce78bcb22df"}
Sep 18 19:17:09.880188 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:09.880118    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2"]
Sep 18 19:17:09.882429 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:09.882398    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4283428_426d_4c47_aa9f_8533caee3298.slice/crio-b4fc3289f3f0dea26c12cb36999f233bbac693e145c33d8190e4060a880c6b97 WatchSource:0}: Error finding container b4fc3289f3f0dea26c12cb36999f233bbac693e145c33d8190e4060a880c6b97: Status 404 returned error can't find the container with id b4fc3289f3f0dea26c12cb36999f233bbac693e145c33d8190e4060a880c6b97
Sep 18 19:17:10.359053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:10.358960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gzmkc" event={"ID":"1215df9a-994e-45a1-86ec-9c48436b5718","Type":"ContainerStarted","Data":"3ea9bb30168cea0de35db6795b5dcb8aa251ede57638cc1e27ef34db647c9f39"}
Sep 18 19:17:10.360168 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:10.360144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2" event={"ID":"c4283428-426d-4c47-aa9f-8533caee3298","Type":"ContainerStarted","Data":"b4fc3289f3f0dea26c12cb36999f233bbac693e145c33d8190e4060a880c6b97"}
Sep 18 19:17:10.386999 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:10.386943    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gzmkc" podStartSLOduration=34.327514387 podStartE2EDuration="36.38692546s" totalImagesPullingTime="2.059411073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:16:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:10.386812646 +0000 UTC m=+162.762238938" watchObservedRunningTime="2026-09-18 19:17:10.38692546 +0000 UTC m=+162.762351749"
Sep 18 19:17:11.365972 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.365882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2" event={"ID":"c4283428-426d-4c47-aa9f-8533caee3298","Type":"ContainerStarted","Data":"6a4d58836cbfd4d1293bba9d72b71b492391a7c34e88d7d0d72c68bac14a2c35"}
Sep 18 19:17:11.365972 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.365937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2" event={"ID":"c4283428-426d-4c47-aa9f-8533caee3298","Type":"ContainerStarted","Data":"ae6b1ef61fe6acd8c51e51429943bd951042e702615cf6e00eece937e5aace78"}
Sep 18 19:17:11.395986 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.395744    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-xttd2" podStartSLOduration=2.09578936 podStartE2EDuration="3.395722896s" totalImagesPullingTime="1.299933536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:11.394445665 +0000 UTC m=+163.769871958" watchObservedRunningTime="2026-09-18 19:17:11.395722896 +0000 UTC m=+163.771149191"
Sep 18 19:17:11.786571 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.786534    2583 scope.go:117] "RemoveContainer" containerID="ba8e3b6223211179e17f0e81c4798021cfb0bb8d345bcc6e6f00b343a9319efc"
Sep 18 19:17:11.973567 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.973528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:11.975939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:11.975916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22a1f40d-14f0-468d-83f8-d34b818e84ee-metrics-tls\") pod \"dns-default-4g8h7\" (UID: \"22a1f40d-14f0-468d-83f8-d34b818e84ee\") " pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:12.003813 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.003786    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:17:12.140087 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.140012    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6m5tz\""
Sep 18 19:17:12.147695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.147679    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:12.371309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.371278    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:17:12.371733 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.371345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" event={"ID":"06e130ed-b5d1-47e6-ad81-26d0cc582900","Type":"ContainerStarted","Data":"cb9a18429c91c71e6c9a6ed775dea387f3fbc210f480f428d7c81acfe433fc74"}
Sep 18 19:17:12.371847 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.371826    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:17:12.397674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.397560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz" podStartSLOduration=40.507265968 podStartE2EDuration="43.397542295s" totalImagesPullingTime="2.890276327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:12.397084346 +0000 UTC m=+164.772510640" watchObservedRunningTime="2026-09-18 19:17:12.397542295 +0000 UTC m=+164.772968589"
Sep 18 19:17:12.427530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.427493    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4g8h7"]
Sep 18 19:17:12.431800 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:12.431758    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a1f40d_14f0_468d_83f8_d34b818e84ee.slice/crio-85469d4ce539256c2cc199491646efadac82fe731d62b0bcac406a10f3c590f9 WatchSource:0}: Error finding container 85469d4ce539256c2cc199491646efadac82fe731d62b0bcac406a10f3c590f9: Status 404 returned error can't find the container with id 85469d4ce539256c2cc199491646efadac82fe731d62b0bcac406a10f3c590f9
Sep 18 19:17:12.452915 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:12.452880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-wbklz"
Sep 18 19:17:13.376771 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.376738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4g8h7" event={"ID":"22a1f40d-14f0-468d-83f8-d34b818e84ee","Type":"ContainerStarted","Data":"85469d4ce539256c2cc199491646efadac82fe731d62b0bcac406a10f3c590f9"}
Sep 18 19:17:13.773278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.773241    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6r87q"]
Sep 18 19:17:13.776124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.776103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.779298 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.779265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:17:13.779443 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.779333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2n9jz\""
Sep 18 19:17:13.779512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.779452    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:17:13.780768 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.780750    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:17:13.790069 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-textfile\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-metrics-client-ca\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59f9\" (UniqueName: \"kubernetes.io/projected/4f027814-71ea-46c3-a90c-91550410847b-kube-api-access-k59f9\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-sys\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-tls\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790395    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-root\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.790479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.790467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-wtmp\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.891810 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-root\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-wtmp\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-textfile\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-metrics-client-ca\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-root\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59f9\" (UniqueName: \"kubernetes.io/projected/4f027814-71ea-46c3-a90c-91550410847b-kube-api-access-k59f9\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.891990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-sys\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892010 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-tls\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892544 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892544 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-wtmp\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892544 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f027814-71ea-46c3-a90c-91550410847b-sys\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892544 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-textfile\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.892750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.892708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f027814-71ea-46c3-a90c-91550410847b-metrics-client-ca\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.894970 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.894944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.895144 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.895119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f027814-71ea-46c3-a90c-91550410847b-node-exporter-tls\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:13.904611 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:13.904584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59f9\" (UniqueName: \"kubernetes.io/projected/4f027814-71ea-46c3-a90c-91550410847b-kube-api-access-k59f9\") pod \"node-exporter-6r87q\" (UID: \"4f027814-71ea-46c3-a90c-91550410847b\") " pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:14.087433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.087397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6r87q"
Sep 18 19:17:14.102889 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:14.102862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f027814_71ea_46c3_a90c_91550410847b.slice/crio-902615feaf2319083b4bb55e5e9f25645c9b85b004b4b713ef237480e1b682f6 WatchSource:0}: Error finding container 902615feaf2319083b4bb55e5e9f25645c9b85b004b4b713ef237480e1b682f6: Status 404 returned error can't find the container with id 902615feaf2319083b4bb55e5e9f25645c9b85b004b4b713ef237480e1b682f6
Sep 18 19:17:14.382585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.382485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4g8h7" event={"ID":"22a1f40d-14f0-468d-83f8-d34b818e84ee","Type":"ContainerStarted","Data":"e2e3ea49cbd3dcc8d66398c02ec8221074b59eefadbac44a801221cacf3d692f"}
Sep 18 19:17:14.382585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.382550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4g8h7" event={"ID":"22a1f40d-14f0-468d-83f8-d34b818e84ee","Type":"ContainerStarted","Data":"0485b83f94feb3ccc60a3f5b3f3f671b554f741198ab976bebd694044c9be3f7"}
Sep 18 19:17:14.383041 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.382597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:14.383725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.383705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6r87q" event={"ID":"4f027814-71ea-46c3-a90c-91550410847b","Type":"ContainerStarted","Data":"902615feaf2319083b4bb55e5e9f25645c9b85b004b4b713ef237480e1b682f6"}
Sep 18 19:17:14.421611 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:14.421556    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4g8h7" podStartSLOduration=128.853182852 podStartE2EDuration="2m10.421537632s" totalImagesPullingTime="1.56835478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:14.420258697 +0000 UTC m=+166.795684990" watchObservedRunningTime="2026-09-18 19:17:14.421537632 +0000 UTC m=+166.796963982"
Sep 18 19:17:15.394512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:15.394418    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f027814-71ea-46c3-a90c-91550410847b" containerID="1cc62f12c9291e9b05e2e42c459a5a813f473466beedcdd8c732e5f8fcfe30d2" exitCode=0
Sep 18 19:17:15.394923 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:15.394506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6r87q" event={"ID":"4f027814-71ea-46c3-a90c-91550410847b","Type":"ContainerDied","Data":"1cc62f12c9291e9b05e2e42c459a5a813f473466beedcdd8c732e5f8fcfe30d2"}
Sep 18 19:17:16.400817 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.400778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6r87q" event={"ID":"4f027814-71ea-46c3-a90c-91550410847b","Type":"ContainerStarted","Data":"f3400fd74d61a731cc4c1105e52a2ad99dc5a96e880f608929d00b0f2f1ef495"}
Sep 18 19:17:16.401508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.400846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6r87q" event={"ID":"4f027814-71ea-46c3-a90c-91550410847b","Type":"ContainerStarted","Data":"9e5085c248196b4da1420d32ac77d37e6a9ea32476369ce52f784fc44c8618d0"}
Sep 18 19:17:16.454895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.454823    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6r87q" podStartSLOduration=2.365364592 podStartE2EDuration="3.45480494s" totalImagesPullingTime="1.018426394s" totalInitContainerRuntime="71.013954ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:16.453000322 +0000 UTC m=+168.828426615" watchObservedRunningTime="2026-09-18 19:17:16.45480494 +0000 UTC m=+168.830231234"
Sep 18 19:17:16.776661 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.776623    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"]
Sep 18 19:17:16.779305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.779288    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.790113 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.790087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:17:16.790291 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.790277    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q2fxb\""
Sep 18 19:17:16.791130 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.791113    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:17:16.792936 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.792916    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:17:16.797716 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.797700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9hajpb659lue4\""
Sep 18 19:17:16.803657 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.803639    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:17:16.803840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.803824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:17:16.812442 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.812411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"]
Sep 18 19:17:16.816427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m6x\" (UniqueName: \"kubernetes.io/projected/0ce461fc-a0e5-44c2-b013-31ea73f469e1-kube-api-access-j8m6x\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-grpc-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.816809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.816787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ce461fc-a0e5-44c2-b013-31ea73f469e1-metrics-client-ca\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.917990 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.917954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m6x\" (UniqueName: \"kubernetes.io/projected/0ce461fc-a0e5-44c2-b013-31ea73f469e1-kube-api-access-j8m6x\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-grpc-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ce461fc-a0e5-44c2-b013-31ea73f469e1-metrics-client-ca\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.918517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.918352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.919101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.919073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ce461fc-a0e5-44c2-b013-31ea73f469e1-metrics-client-ca\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921548 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921356    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921548 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.921793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.921771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ce461fc-a0e5-44c2-b013-31ea73f469e1-secret-grpc-tls\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:16.959251 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:16.959220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m6x\" (UniqueName: \"kubernetes.io/projected/0ce461fc-a0e5-44c2-b013-31ea73f469e1-kube-api-access-j8m6x\") pod \"thanos-querier-848f7dcf97-xvbsb\" (UID: \"0ce461fc-a0e5-44c2-b013-31ea73f469e1\") " pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:17.089607 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:17.089511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:17.410264 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:17.410192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"]
Sep 18 19:17:17.413568 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:17.413546    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce461fc_a0e5_44c2_b013_31ea73f469e1.slice/crio-1a5b869af9f5ae3bff39576fbf94cfe45b2d5947d5734763583bdf42ed4263cc WatchSource:0}: Error finding container 1a5b869af9f5ae3bff39576fbf94cfe45b2d5947d5734763583bdf42ed4263cc: Status 404 returned error can't find the container with id 1a5b869af9f5ae3bff39576fbf94cfe45b2d5947d5734763583bdf42ed4263cc
Sep 18 19:17:18.185006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.184967    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"]
Sep 18 19:17:18.187712 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.187691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.192397 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.192355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5hsmp\""
Sep 18 19:17:18.192772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.192753    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 19:17:18.226857 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.226793    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"]
Sep 18 19:17:18.228369 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.228334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htt4f\" (UniqueName: \"kubernetes.io/projected/25ed0475-e04e-4064-939f-19eafd7748f0-kube-api-access-htt4f\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.228522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.228418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.329025 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.328984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.329222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.329082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htt4f\" (UniqueName: \"kubernetes.io/projected/25ed0475-e04e-4064-939f-19eafd7748f0-kube-api-access-htt4f\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.329222 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:18.329145    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 19:17:18.329328 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:18.329223    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert podName:25ed0475-e04e-4064-939f-19eafd7748f0 nodeName:}" failed. No retries permitted until 2026-09-18 19:17:18.82919828 +0000 UTC m=+171.204624551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-b9zds" (UID: "25ed0475-e04e-4064-939f-19eafd7748f0") : secret "monitoring-plugin-cert" not found
Sep 18 19:17:18.355623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.355585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htt4f\" (UniqueName: \"kubernetes.io/projected/25ed0475-e04e-4064-939f-19eafd7748f0-kube-api-access-htt4f\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.415568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.415518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"1a5b869af9f5ae3bff39576fbf94cfe45b2d5947d5734763583bdf42ed4263cc"}
Sep 18 19:17:18.832725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.832687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:18.835515 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:18.835479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25ed0475-e04e-4064-939f-19eafd7748f0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-b9zds\" (UID: \"25ed0475-e04e-4064-939f-19eafd7748f0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:19.099194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:19.099096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:19.421649 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:19.421619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"2e9affe8cc83ae8c218a0ead98a1f6b145ac6e7fb73bd29db6403c3007d03575"}
Sep 18 19:17:19.422035 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:19.421681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"785803c0e475570c59c78f8423e224b60e8e5ba82abd9f12ace1ee7eb09db262"}
Sep 18 19:17:19.422035 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:19.421713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"0943f9d13c63515ba13d03c0ed071a3ea71394b17e2c40fbed80beb442013180"}
Sep 18 19:17:19.426374 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:19.426335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"]
Sep 18 19:17:19.429531 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:19.429496    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ed0475_e04e_4064_939f_19eafd7748f0.slice/crio-b2e38ab117cf5f10e438b0c860f2826489bc61a24d887135ce97e570860af025 WatchSource:0}: Error finding container b2e38ab117cf5f10e438b0c860f2826489bc61a24d887135ce97e570860af025: Status 404 returned error can't find the container with id b2e38ab117cf5f10e438b0c860f2826489bc61a24d887135ce97e570860af025
Sep 18 19:17:20.429595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.429495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"5992e479312a17a99b9efa7e2a1954915912336fe28275729e8c7b21c521a0b9"}
Sep 18 19:17:20.429595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.429582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"1f8610e96ef4c5dad62cac0c187e4a656594f4489e7479a59749142f50360ad6"}
Sep 18 19:17:20.430009 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.429631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:20.430009 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.429667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" event={"ID":"0ce461fc-a0e5-44c2-b013-31ea73f469e1","Type":"ContainerStarted","Data":"90362d410db8eb022bd1d36167df49b955888aa4f9e89a345a16b55886b64f48"}
Sep 18 19:17:20.431122 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.431088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds" event={"ID":"25ed0475-e04e-4064-939f-19eafd7748f0","Type":"ContainerStarted","Data":"b2e38ab117cf5f10e438b0c860f2826489bc61a24d887135ce97e570860af025"}
Sep 18 19:17:20.494557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.494503    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb" podStartSLOduration=2.015461253 podStartE2EDuration="4.494484338s" totalImagesPullingTime="2.479023085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:20.492387738 +0000 UTC m=+172.867814032" watchObservedRunningTime="2026-09-18 19:17:20.494484338 +0000 UTC m=+172.869910631"
Sep 18 19:17:20.661559 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.660990    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:20.666916 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.666889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.671491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.671460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:17:20.671695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.671671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:17:20.671785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.671675    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:17:20.671938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.671914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8gh2s\""
Sep 18 19:17:20.672041 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.671915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:17:20.672125 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.672052    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:17:20.672182 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.672131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.672866    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.672887    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673028    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1nlcetircgac0\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673122    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673141    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:17:20.673327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:17:20.673742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.673519    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:17:20.691450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.691128    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:17:20.693174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.693137    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:17:20.694190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.694049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:20.752056 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752202 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752202 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752202 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752306    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9xv\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752514 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752514 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752514 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.752796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.752713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855209 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855342 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855342 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9xv\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855628 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.855906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.856269 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.855984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.856903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.856594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.857020 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.856923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.858326 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.857996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.858535 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.858508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.859392 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.859156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.859877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.859853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.860523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.860422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.861835 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.861158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.861835 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.861392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.861835 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.861513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.861835 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.861585    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.862089 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.861882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.862656 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.862594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.862656 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.862643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.863653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.863529    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.863653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.863610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.863800 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.863772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.865833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.865809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.867936 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.867913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9xv\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv\") pod \"prometheus-k8s-0\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:20.980556 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:20.980227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:21.279096 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.279071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:17:21.281265 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:17:21.281234    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98de7a55_9a65_4715_808e_5a63b34cd95c.slice/crio-5bf813f6916c2af6147a9b1c2f36d2110218b6336fe53616b9376abf2960ed04 WatchSource:0}: Error finding container 5bf813f6916c2af6147a9b1c2f36d2110218b6336fe53616b9376abf2960ed04: Status 404 returned error can't find the container with id 5bf813f6916c2af6147a9b1c2f36d2110218b6336fe53616b9376abf2960ed04
Sep 18 19:17:21.435846 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.435801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"5bf813f6916c2af6147a9b1c2f36d2110218b6336fe53616b9376abf2960ed04"}
Sep 18 19:17:21.437311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.437282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds" event={"ID":"25ed0475-e04e-4064-939f-19eafd7748f0","Type":"ContainerStarted","Data":"c97976f406e57c3648e2b94a13958a7e1bd3f900ff83be5aadb52a78e051c15f"}
Sep 18 19:17:21.437684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.437667    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:21.442259 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.442239    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds"
Sep 18 19:17:21.460282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:21.460235    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-b9zds" podStartSLOduration=2.096669803 podStartE2EDuration="3.46021915s" totalImagesPullingTime="1.363549347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:21.459914291 +0000 UTC m=+173.835340580" watchObservedRunningTime="2026-09-18 19:17:21.46021915 +0000 UTC m=+173.835645442"
Sep 18 19:17:22.786290 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:22.786260    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:17:23.446827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:23.446794    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff" exitCode=0
Sep 18 19:17:23.446975 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:23.446876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"}
Sep 18 19:17:24.397143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:24.397113    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4g8h7"
Sep 18 19:17:26.444664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:26.444636    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-848f7dcf97-xvbsb"
Sep 18 19:17:27.020583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.020535    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerName="registry" containerID="cri-o://66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c" gracePeriod=30
Sep 18 19:17:27.263389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.263350    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:17:27.315118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315126    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxtl9\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315180    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315492 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.315596 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates\") pod \"9540b991-d8d3-4381-a4e7-576181a4cb79\" (UID: \"9540b991-d8d3-4381-a4e7-576181a4cb79\") "
Sep 18 19:17:27.316124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:27.316124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.315870    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-trusted-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.316302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.316119    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:17:27.317646 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.317620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9" (OuterVolumeSpecName: "kube-api-access-nxtl9") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "kube-api-access-nxtl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:27.317777 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.317727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:27.317777 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.317756    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:27.317858 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.317828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:17:27.318180 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.318149    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:17:27.323794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.323771    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9540b991-d8d3-4381-a4e7-576181a4cb79" (UID: "9540b991-d8d3-4381-a4e7-576181a4cb79"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:17:27.416226 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416205    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9540b991-d8d3-4381-a4e7-576181a4cb79-ca-trust-extracted\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416226 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416223    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416232    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-image-registry-private-configuration\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416241    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9540b991-d8d3-4381-a4e7-576181a4cb79-registry-certificates\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416251    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9540b991-d8d3-4381-a4e7-576181a4cb79-installation-pull-secrets\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416259    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxtl9\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-kube-api-access-nxtl9\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.416426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.416267    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9540b991-d8d3-4381-a4e7-576181a4cb79-bound-sa-token\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:17:27.464845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.464820    2583 generic.go:356] "Generic (PLEG): container finished" podID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerID="66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c" exitCode=0
Sep 18 19:17:27.465193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.464873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4"
Sep 18 19:17:27.465193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.464919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" event={"ID":"9540b991-d8d3-4381-a4e7-576181a4cb79","Type":"ContainerDied","Data":"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"}
Sep 18 19:17:27.465193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.464979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54c64c4d7-hscg4" event={"ID":"9540b991-d8d3-4381-a4e7-576181a4cb79","Type":"ContainerDied","Data":"fca84956ef2bdb70ca9c143f9bc7680e521c1b9522d24fc15a55f7099a5d31a9"}
Sep 18 19:17:27.465193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.465010    2583 scope.go:117] "RemoveContainer" containerID="66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"
Sep 18 19:17:27.469317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"}
Sep 18 19:17:27.469433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"}
Sep 18 19:17:27.469506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"}
Sep 18 19:17:27.469506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"}
Sep 18 19:17:27.469615 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"}
Sep 18 19:17:27.469615 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.469558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerStarted","Data":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"}
Sep 18 19:17:27.474502 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.474482    2583 scope.go:117] "RemoveContainer" containerID="66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"
Sep 18 19:17:27.474785 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:17:27.474765    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c\": container with ID starting with 66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c not found: ID does not exist" containerID="66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"
Sep 18 19:17:27.474835 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.474794    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c"} err="failed to get container status \"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c\": rpc error: code = NotFound desc = could not find container \"66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c\": container with ID starting with 66b7624df3b50cd76d25f2374a377682dde3c2a9a008c6253dd5c6dd71ecaa1c not found: ID does not exist"
Sep 18 19:17:27.508956 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.508881    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.959540992 podStartE2EDuration="7.508868109s" totalImagesPullingTime="4.527323228s" totalInitContainerRuntime="22.003889ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:17:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:17:27.506209186 +0000 UTC m=+179.881635492" watchObservedRunningTime="2026-09-18 19:17:27.508868109 +0000 UTC m=+179.884294402"
Sep 18 19:17:27.526116 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.526093    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:17:27.530789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.530770    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54c64c4d7-hscg4"]
Sep 18 19:17:27.791632 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:27.791561    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" path="/var/lib/kubelet/pods/9540b991-d8d3-4381-a4e7-576181a4cb79/volumes"
Sep 18 19:17:30.980975 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:30.980935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:17:39.520910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:39.520874    2583 generic.go:356] "Generic (PLEG): container finished" podID="8831daaf-6e04-4e49-bc7d-9e5de158a32c" containerID="471eb48fa22f865716b8e52a4cff2e23da3340f7c406a10e612aa998d73d45ce" exitCode=0
Sep 18 19:17:39.521347 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:39.520952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25" event={"ID":"8831daaf-6e04-4e49-bc7d-9e5de158a32c","Type":"ContainerDied","Data":"471eb48fa22f865716b8e52a4cff2e23da3340f7c406a10e612aa998d73d45ce"}
Sep 18 19:17:39.521347 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:39.521278    2583 scope.go:117] "RemoveContainer" containerID="471eb48fa22f865716b8e52a4cff2e23da3340f7c406a10e612aa998d73d45ce"
Sep 18 19:17:40.531081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:40.531044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-8tm25" event={"ID":"8831daaf-6e04-4e49-bc7d-9e5de158a32c","Type":"ContainerStarted","Data":"f3fab23149d01e860fe68744a5a674bac5b38c6f0b7862cd2f4019e319f2cf1d"}
Sep 18 19:17:49.568351 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:49.568318    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8ae4017-1968-4419-8996-91642ebd40ae" containerID="bc1ecb4de8f1896219c8bfba6a3675ad414f079952561b26bc1fe72961589a00" exitCode=0
Sep 18 19:17:49.568824 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:49.568382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f" event={"ID":"c8ae4017-1968-4419-8996-91642ebd40ae","Type":"ContainerDied","Data":"bc1ecb4de8f1896219c8bfba6a3675ad414f079952561b26bc1fe72961589a00"}
Sep 18 19:17:49.568824 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:49.568744    2583 scope.go:117] "RemoveContainer" containerID="bc1ecb4de8f1896219c8bfba6a3675ad414f079952561b26bc1fe72961589a00"
Sep 18 19:17:50.575735 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:17:50.575695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-9kp6f" event={"ID":"c8ae4017-1968-4419-8996-91642ebd40ae","Type":"ContainerStarted","Data":"6a49485aaa5c82b48987685d913ca3eecb5faf8ff8afd7ba1f58c46561df2deb"}
Sep 18 19:18:20.981273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:20.981246    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:20.996980 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:20.996959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:21.720568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:21.720539    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:39.505223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505178    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505588    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="prometheus" containerID="cri-o://59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16" gracePeriod=600
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505608    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="thanos-sidecar" containerID="cri-o://a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2" gracePeriod=600
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505608    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy" containerID="cri-o://350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2" gracePeriod=600
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505621    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-web" containerID="cri-o://6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3" gracePeriod=600
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505675    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="config-reloader" containerID="cri-o://6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2" gracePeriod=600
Sep 18 19:18:39.505774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.505727    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7" gracePeriod=600
Sep 18 19:18:39.748915 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.748886    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785020    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785053    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785062    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785070    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785077    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785084    2583 generic.go:356] "Generic (PLEG): container finished" podID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16" exitCode=0
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785100    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:39.785123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785285    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785291    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"}
Sep 18 19:18:39.785623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.785498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"98de7a55-9a65-4715-808e-5a63b34cd95c","Type":"ContainerDied","Data":"5bf813f6916c2af6147a9b1c2f36d2110218b6336fe53616b9376abf2960ed04"}
Sep 18 19:18:39.795406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.795378    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.809872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.809849    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.821924 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.821766    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.831639 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.831544    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.840069 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.840042    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.853622 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.852869    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.855854 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.855828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.855933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.855877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.855933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.855926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856027 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.855970    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856027 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856218 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856218 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h9xv\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856218 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.856389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.856840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.857286 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.857353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.857592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857671 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857733 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857733 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.857820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.857820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857774    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.858003 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.858003 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.857871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls\") pod \"98de7a55-9a65-4715-808e-5a63b34cd95c\" (UID: \"98de7a55-9a65-4715-808e-5a63b34cd95c\") "
Sep 18 19:18:39.858165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858147    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.858228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858172    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.858228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858189    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.858228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858205    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.858228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858220    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-configmap-metrics-client-ca\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.859116 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.858857    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.859383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.859346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:18:39.861629 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.860351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:18:39.862182 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.862092    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.863711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.863667    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.863908 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.863869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.864257 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864227    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.864442 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864411    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:18:39.864730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864627    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out" (OuterVolumeSpecName: "config-out") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:18:39.864730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config" (OuterVolumeSpecName: "config") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.864730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864658    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.864730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864689    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.864730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv" (OuterVolumeSpecName: "kube-api-access-5h9xv") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "kube-api-access-5h9xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:18:39.865033 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.864764    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.865033 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864823    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.865033 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.864853    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.865238 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.865200    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.865340 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.865237    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.865340 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.865262    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.865651 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.865625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.865745 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.865703    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.865806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.865734    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.865806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.865756    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.866085 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.866065    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.866143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866090    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.866143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866106    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.866269 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.866524 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.866411    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.866524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866440    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.866524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866457    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.866852 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.866834    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.866935 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866858    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.866935 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.866876    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.867175 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:18:39.867153    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.867230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867182    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.867230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867200    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.867506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867486    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.867506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867504    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.867821 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867786    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.867894 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.867823    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.868095 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868075    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.868168 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868096    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.868450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868340    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.868450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868400    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.868684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868659    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.868742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.868687    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.871501 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.871479    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.871501 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.871500    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.871833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.871814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.871918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.871837    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.872174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872144    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.872422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872175    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.872552 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872526    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.872617 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872554    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.872859 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872834    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.872926 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.872860    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.873513 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.873490    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.873619 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.873513    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.873802 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.873782    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.873881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.873803    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.874199 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874145    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.874199 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874172    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.874645 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874603    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.874645 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874625    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.874949 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874924    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.875034 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.874950    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.875688 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.875638    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.875780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.875688    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.875981 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.875963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.876049 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.875981    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.876204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876186    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.876448 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876204    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.876527 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.876527 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876498    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.876815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876785    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.876815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.876816    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.877095 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877076    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.877173 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877096    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.877384 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877350    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.877384 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877383    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.877692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877669    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.877793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877695    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.877979 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877959    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.878052 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.877981    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.878213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878194    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config" (OuterVolumeSpecName: "web-config") pod "98de7a55-9a65-4715-808e-5a63b34cd95c" (UID: "98de7a55-9a65-4715-808e-5a63b34cd95c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:18:39.878282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878194    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.878282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878239    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.878487 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878469    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.878564 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878488    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.878705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878687    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.878750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878706    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.878906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878889    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.878956 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.878906    2583 scope.go:117] "RemoveContainer" containerID="bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"
Sep 18 19:18:39.879160 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879144    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7"} err="failed to get container status \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": rpc error: code = NotFound desc = could not find container \"bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7\": container with ID starting with bd5471c3a69812809ee0bbff898a0fc81d09aa2defbdf94c99c2b2c959ddc5a7 not found: ID does not exist"
Sep 18 19:18:39.879207 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879162    2583 scope.go:117] "RemoveContainer" containerID="350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"
Sep 18 19:18:39.879402 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879349    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2"} err="failed to get container status \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": rpc error: code = NotFound desc = could not find container \"350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2\": container with ID starting with 350ea7dfff0cf9885ec98ecf5c6dd5b91e3f5ac07267facb534a09c42a4421d2 not found: ID does not exist"
Sep 18 19:18:39.879491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879403    2583 scope.go:117] "RemoveContainer" containerID="6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"
Sep 18 19:18:39.879646 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879629    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3"} err="failed to get container status \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": rpc error: code = NotFound desc = could not find container \"6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3\": container with ID starting with 6f2abf0df22f509175887e796104e7b51a78868c412f8a0cf76b2356b05975f3 not found: ID does not exist"
Sep 18 19:18:39.879689 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879647    2583 scope.go:117] "RemoveContainer" containerID="a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"
Sep 18 19:18:39.879870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879852    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2"} err="failed to get container status \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": rpc error: code = NotFound desc = could not find container \"a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2\": container with ID starting with a500fc6a38bfcb83c3e658c5cf48d735548499dd39c1a1d28d185c3e12249cd2 not found: ID does not exist"
Sep 18 19:18:39.879925 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.879870    2583 scope.go:117] "RemoveContainer" containerID="6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"
Sep 18 19:18:39.880079 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.880062    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2"} err="failed to get container status \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": rpc error: code = NotFound desc = could not find container \"6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2\": container with ID starting with 6eb3d700dffe30ee272599e7d505b64330da7391d5ffb2c82378bae2346527c2 not found: ID does not exist"
Sep 18 19:18:39.880131 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.880079    2583 scope.go:117] "RemoveContainer" containerID="59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"
Sep 18 19:18:39.880307 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.880290    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16"} err="failed to get container status \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": rpc error: code = NotFound desc = could not find container \"59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16\": container with ID starting with 59193c2202044022e15413b1990a81f24947b205d4eb6becdeaf3538ccf10e16 not found: ID does not exist"
Sep 18 19:18:39.880307 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.880307    2583 scope.go:117] "RemoveContainer" containerID="2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"
Sep 18 19:18:39.880562 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.880546    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff"} err="failed to get container status \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": rpc error: code = NotFound desc = could not find container \"2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff\": container with ID starting with 2c8b577fa2c7478d051104062aa7c830d441dad015cc858808c4403a5db773ff not found: ID does not exist"
Sep 18 19:18:39.959436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959405    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-config-out\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959431    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-kube-rbac-proxy\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959436 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959454    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959465    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959475    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9xv\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-kube-api-access-5h9xv\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959484    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/98de7a55-9a65-4715-808e-5a63b34cd95c-tls-assets\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959492    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959500    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959509    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959520    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-web-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959529    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959537    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-metrics-client-certs\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959547    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/98de7a55-9a65-4715-808e-5a63b34cd95c-prometheus-k8s-db\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959556    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-secret-grpc-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:39.959683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:39.959563    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/98de7a55-9a65-4715-808e-5a63b34cd95c-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:18:40.134542 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.134505    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:40.135723 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.135700    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:40.200529 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200499    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:40.200790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="thanos-sidecar"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="thanos-sidecar"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200802    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200807    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200815    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerName="registry"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200820    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerName="registry"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200829    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="config-reloader"
Sep 18 19:18:40.200833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200834    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="config-reloader"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200844    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="init-config-reloader"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200849    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="init-config-reloader"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200854    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-web"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-web"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200870    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="prometheus"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200875    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="prometheus"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200880    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-thanos"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200885    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-thanos"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200927    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="thanos-sidecar"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200934    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="prometheus"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200940    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-thanos"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200945    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9540b991-d8d3-4381-a4e7-576181a4cb79" containerName="registry"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200952    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy-web"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200964    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="config-reloader"
Sep 18 19:18:40.201058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.200973    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" containerName="kube-rbac-proxy"
Sep 18 19:18:40.204678 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.204660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.225326 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.225297    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 19:18:40.229739 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.229719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 19:18:40.238215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 19:18:40.238346 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238329    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 19:18:40.238422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238329    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 19:18:40.238422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238329    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 19:18:40.238534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 19:18:40.238587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 19:18:40.238640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 19:18:40.238691 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 19:18:40.238742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 19:18:40.238742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238721    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 19:18:40.238901 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.238884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8gh2s\""
Sep 18 19:18:40.239497 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.239483    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 19:18:40.239726 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.239709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1nlcetircgac0\""
Sep 18 19:18:40.245263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.245245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 19:18:40.254659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.254637    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 19:18:40.262271 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.262206    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:40.363488 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363488 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363488 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-config-out\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-web-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.363772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdcg\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-kube-api-access-gtdcg\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.364062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.363944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdcg\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-kube-api-access-gtdcg\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.464747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464824    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.464942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-config-out\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-web-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.465806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.465685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.466840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.466079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.466840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.466536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.466840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.466766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.468460 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.467750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.468460 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.468409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.468655 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.468599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.468718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.468653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96ea603c-cafe-4dbc-a915-ba779db8b4da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.469505 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.469477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.469720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96ea603c-cafe-4dbc-a915-ba779db8b4da-config-out\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.469961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-web-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.469977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-config\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.470059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470329 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.470179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470329 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.470244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.470531    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.470618 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.470603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96ea603c-cafe-4dbc-a915-ba779db8b4da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.478556 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.478539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdcg\" (UniqueName: \"kubernetes.io/projected/96ea603c-cafe-4dbc-a915-ba779db8b4da-kube-api-access-gtdcg\") pod \"prometheus-k8s-0\" (UID: \"96ea603c-cafe-4dbc-a915-ba779db8b4da\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.514118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.514081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:18:40.831228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:40.831182    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 19:18:40.835177 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:18:40.835147    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ea603c_cafe_4dbc_a915_ba779db8b4da.slice/crio-9ec8838ab9c833cba58548a8b133e5783f29dc8504bea836538aff9a0c76323a WatchSource:0}: Error finding container 9ec8838ab9c833cba58548a8b133e5783f29dc8504bea836538aff9a0c76323a: Status 404 returned error can't find the container with id 9ec8838ab9c833cba58548a8b133e5783f29dc8504bea836538aff9a0c76323a
Sep 18 19:18:41.675135 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.675098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:18:41.677420 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.677398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04-metrics-certs\") pod \"network-metrics-daemon-nsgdm\" (UID: \"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04\") " pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:18:41.692107 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.692082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xjcwp\""
Sep 18 19:18:41.698167 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.698151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nsgdm"
Sep 18 19:18:41.791902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.791867    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98de7a55-9a65-4715-808e-5a63b34cd95c" path="/var/lib/kubelet/pods/98de7a55-9a65-4715-808e-5a63b34cd95c/volumes"
Sep 18 19:18:41.793869 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.793840    2583 generic.go:356] "Generic (PLEG): container finished" podID="96ea603c-cafe-4dbc-a915-ba779db8b4da" containerID="4f35037f1cf067d7cf77ad7c52828cce315ef42c4c675615dd45f77f4e8c4b40" exitCode=0
Sep 18 19:18:41.794021 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.793884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerDied","Data":"4f35037f1cf067d7cf77ad7c52828cce315ef42c4c675615dd45f77f4e8c4b40"}
Sep 18 19:18:41.794021 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.793948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"9ec8838ab9c833cba58548a8b133e5783f29dc8504bea836538aff9a0c76323a"}
Sep 18 19:18:41.973444 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:41.973398    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nsgdm"]
Sep 18 19:18:41.977648 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:18:41.977497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6e644a_9450_4e4a_b6d4_1f9cbcc94a04.slice/crio-45fcb82568bff477b77c951974976a2f882a8da70b2782bb221387326ccf1bbf WatchSource:0}: Error finding container 45fcb82568bff477b77c951974976a2f882a8da70b2782bb221387326ccf1bbf: Status 404 returned error can't find the container with id 45fcb82568bff477b77c951974976a2f882a8da70b2782bb221387326ccf1bbf
Sep 18 19:18:42.800598 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.800558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsgdm" event={"ID":"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04","Type":"ContainerStarted","Data":"45fcb82568bff477b77c951974976a2f882a8da70b2782bb221387326ccf1bbf"}
Sep 18 19:18:42.804985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.804940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"796b3e40498d18cd642388145bfb38ef48a0428c3cd1dfbd40121c4ebc6e1869"}
Sep 18 19:18:42.805169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.805031    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"3ac05bec82fe6b0bd41fbab9b0d6844b9dcfa5f9a21a1048a21983b81a83a371"}
Sep 18 19:18:42.805169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.805079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"c00f44be33b5f47442308517f8c79679263fb0a3f978370d02c5a8252c81452e"}
Sep 18 19:18:42.805169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.805119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"8e41390d13157ca23bfb2efeda84b6408cd768970bd5196926fcca27e2d41738"}
Sep 18 19:18:42.805169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.805160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"add6cf08973b2a8b8b2f6067cc438bb283eb5b374b7a67ba26c2310408365036"}
Sep 18 19:18:42.805394 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.805194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"96ea603c-cafe-4dbc-a915-ba779db8b4da","Type":"ContainerStarted","Data":"0e93d2acd980d5e9c3591ee51ac3a097e09238e4fafed1fe3eb2216bf73e4cc4"}
Sep 18 19:18:42.852702 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:42.852620    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.836302252 podStartE2EDuration="2.852594174s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.291922ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:42.849520619 +0000 UTC m=+255.224946927" watchObservedRunningTime="2026-09-18 19:18:42.852594174 +0000 UTC m=+255.228020468"
Sep 18 19:18:43.810625 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:43.810586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsgdm" event={"ID":"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04","Type":"ContainerStarted","Data":"c587ce2069687bac22ed7bcabbbf2c5224996df91f5706ae1de0d47b3c277acd"}
Sep 18 19:18:43.811051 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:43.810646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nsgdm" event={"ID":"9f6e644a-9450-4e4a-b6d4-1f9cbcc94a04","Type":"ContainerStarted","Data":"9707102f3d90048a65332414c108693ceb8cdc934f1824009d20d395b729372d"}
Sep 18 19:18:43.848025 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:43.847972    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nsgdm" podStartSLOduration=252.915607017 podStartE2EDuration="4m13.847958365s" totalImagesPullingTime="932.351348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:18:43.847589073 +0000 UTC m=+256.223015378" watchObservedRunningTime="2026-09-18 19:18:43.847958365 +0000 UTC m=+256.223384658"
Sep 18 19:18:45.514784 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:18:45.514747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:19:29.672138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:29.672107    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:19:29.674703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:29.674682    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:19:29.686723 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:29.686702    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:19:40.514644 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:40.514603    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:19:40.530450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:40.530413    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:19:41.054458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:19:41.054430    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 19:23:30.426118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.426040    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-sgdgh"]
Sep 18 19:23:30.429159 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.429143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.432877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.432851    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:23:30.433022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.432881    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:23:30.433022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.432938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:23:30.433768 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.433747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:23:30.434019 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.434004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:23:30.448038 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.448009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-sgdgh"]
Sep 18 19:23:30.472031 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.472001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.472180 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.472049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6lg\" (UniqueName: \"kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.573289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.573253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.573475 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.573303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6lg\" (UniqueName: \"kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.573647 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.573626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.585270 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.585245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6lg\" (UniqueName: \"kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg\") pod \"s3-init-sgdgh\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") " pod="kserve/s3-init-sgdgh"
Sep 18 19:23:30.748899 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:30.748866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sgdgh"
Sep 18 19:23:31.082121 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:31.082031    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-sgdgh"]
Sep 18 19:23:31.085200 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:23:31.085163    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56be5525_3861_4a4e_93eb_036a076af52b.slice/crio-715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac WatchSource:0}: Error finding container 715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac: Status 404 returned error can't find the container with id 715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac
Sep 18 19:23:31.087119 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:31.087100    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:23:31.967767 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:31.967697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerStarted","Data":"715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac"}
Sep 18 19:23:34.983232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:34.983196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerStarted","Data":"bb98fb83e70272992bccaf014b3b36b67ceff3aa1ceac1e49309b05a576f8c52"}
Sep 18 19:23:36.993765 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:36.993728    2583 generic.go:356] "Generic (PLEG): container finished" podID="56be5525-3861-4a4e-93eb-036a076af52b" containerID="bb98fb83e70272992bccaf014b3b36b67ceff3aa1ceac1e49309b05a576f8c52" exitCode=0
Sep 18 19:23:36.994155 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:36.993761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerDied","Data":"bb98fb83e70272992bccaf014b3b36b67ceff3aa1ceac1e49309b05a576f8c52"}
Sep 18 19:23:42.019324 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:42.019285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerStarted","Data":"ea17600ebe33021f9ea6b0fb2c4e05390f6e43a0a18c153a5c1d570e98fb3bef"}
Sep 18 19:23:42.039099 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:23:42.039031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-sgdgh" podStartSLOduration=1.9877508640000001 podStartE2EDuration="12.039015254s" totalImagesPullingTime="8.211789092s" totalInitContainerRuntime="1.839475298s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:23:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:23:42.037192101 +0000 UTC m=+554.412618394" watchObservedRunningTime="2026-09-18 19:23:42.039015254 +0000 UTC m=+554.414441547"
Sep 18 19:24:29.708397 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:29.708354    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:24:29.708956 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:29.708932    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:24:47.285341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:47.285307    2583 generic.go:356] "Generic (PLEG): container finished" podID="56be5525-3861-4a4e-93eb-036a076af52b" containerID="ea17600ebe33021f9ea6b0fb2c4e05390f6e43a0a18c153a5c1d570e98fb3bef" exitCode=0
Sep 18 19:24:47.285767 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:47.285396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerDied","Data":"ea17600ebe33021f9ea6b0fb2c4e05390f6e43a0a18c153a5c1d570e98fb3bef"}
Sep 18 19:24:48.413045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.413009    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sgdgh"
Sep 18 19:24:48.532281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.532235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df6lg\" (UniqueName: \"kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg\") pod \"56be5525-3861-4a4e-93eb-036a076af52b\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") "
Sep 18 19:24:48.532483 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.532320    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache\") pod \"56be5525-3861-4a4e-93eb-036a076af52b\" (UID: \"56be5525-3861-4a4e-93eb-036a076af52b\") "
Sep 18 19:24:48.534471 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.534442    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg" (OuterVolumeSpecName: "kube-api-access-df6lg") pod "56be5525-3861-4a4e-93eb-036a076af52b" (UID: "56be5525-3861-4a4e-93eb-036a076af52b"). InnerVolumeSpecName "kube-api-access-df6lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:24:48.550673 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.550597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache" (OuterVolumeSpecName: "model-cache") pod "56be5525-3861-4a4e-93eb-036a076af52b" (UID: "56be5525-3861-4a4e-93eb-036a076af52b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:24:48.633779 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.633734    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6lg\" (UniqueName: \"kubernetes.io/projected/56be5525-3861-4a4e-93eb-036a076af52b-kube-api-access-df6lg\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:24:48.633779 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:48.633771    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56be5525-3861-4a4e-93eb-036a076af52b-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:24:49.294478 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:49.294441    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-sgdgh"
Sep 18 19:24:49.294641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:49.294474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-sgdgh" event={"ID":"56be5525-3861-4a4e-93eb-036a076af52b","Type":"ContainerDied","Data":"715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac"}
Sep 18 19:24:49.294641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:49.294513    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715a30412595e992fb442c5d4f151a8b6a2ca95a8f80ba39ce573dfa8c930fac"
Sep 18 19:24:56.751418 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751383    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-jj7lb"]
Sep 18 19:24:56.751790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751691    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56be5525-3861-4a4e-93eb-036a076af52b" containerName="download-hf-model"
Sep 18 19:24:56.751790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751704    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56be5525-3861-4a4e-93eb-036a076af52b" containerName="download-hf-model"
Sep 18 19:24:56.751790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751716    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56be5525-3861-4a4e-93eb-036a076af52b" containerName="s3-init"
Sep 18 19:24:56.751790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751723    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56be5525-3861-4a4e-93eb-036a076af52b" containerName="s3-init"
Sep 18 19:24:56.751790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.751784    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="56be5525-3861-4a4e-93eb-036a076af52b" containerName="s3-init"
Sep 18 19:24:56.754917 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.754899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:56.758327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.758303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:24:56.758448 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.758330    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:24:56.758448 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.758304    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 18 19:24:56.758531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.758492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:24:56.759790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.759775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:24:56.765961 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.765936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-jj7lb"]
Sep 18 19:24:56.903836 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.903792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:56.904034 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:56.903894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlq6\" (UniqueName: \"kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.005353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.005262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlq6\" (UniqueName: \"kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.005545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.005423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.005815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.005797    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.015613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.015580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlq6\" (UniqueName: \"kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6\") pod \"s3-tls-init-custom-jj7lb\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") " pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.084116 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.084073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:24:57.361748 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:57.361716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-jj7lb"]
Sep 18 19:24:58.332445 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:58.332402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerStarted","Data":"2800947a8dd17dafcf4893e08b6cbb4fde9534ca3e9559b1876ba87b382d71dc"}
Sep 18 19:24:58.332897 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:58.332454    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerStarted","Data":"fb0e02485d530f558ff60ad05bfbe5c07209fb00c26f3ccee279e75bb83706cc"}
Sep 18 19:24:59.337808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:59.337776    2583 generic.go:356] "Generic (PLEG): container finished" podID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerID="2800947a8dd17dafcf4893e08b6cbb4fde9534ca3e9559b1876ba87b382d71dc" exitCode=0
Sep 18 19:24:59.338211 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:24:59.337871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerDied","Data":"2800947a8dd17dafcf4893e08b6cbb4fde9534ca3e9559b1876ba87b382d71dc"}
Sep 18 19:25:00.344301 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:00.344267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerStarted","Data":"7e76dbc805fb780dc10a0a3e8cf404255d4d31ee81ff8769db8387c30f0c1869"}
Sep 18 19:25:00.373850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:00.373791    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-jj7lb" podStartSLOduration=2.555055615 podStartE2EDuration="4.373769501s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.818713886s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:25:00.371322972 +0000 UTC m=+632.746749270" watchObservedRunningTime="2026-09-18 19:25:00.373769501 +0000 UTC m=+632.749195796"
Sep 18 19:25:07.380120 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:07.380038    2583 generic.go:356] "Generic (PLEG): container finished" podID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerID="7e76dbc805fb780dc10a0a3e8cf404255d4d31ee81ff8769db8387c30f0c1869" exitCode=0
Sep 18 19:25:07.380120 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:07.380095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerDied","Data":"7e76dbc805fb780dc10a0a3e8cf404255d4d31ee81ff8769db8387c30f0c1869"}
Sep 18 19:25:08.531799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.531767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:25:08.604848 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.604810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache\") pod \"2aaba14b-4684-4767-a2f6-1033f8d533cd\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") "
Sep 18 19:25:08.604848 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.604852    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhlq6\" (UniqueName: \"kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6\") pod \"2aaba14b-4684-4767-a2f6-1033f8d533cd\" (UID: \"2aaba14b-4684-4767-a2f6-1033f8d533cd\") "
Sep 18 19:25:08.607134 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.607100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6" (OuterVolumeSpecName: "kube-api-access-mhlq6") pod "2aaba14b-4684-4767-a2f6-1033f8d533cd" (UID: "2aaba14b-4684-4767-a2f6-1033f8d533cd"). InnerVolumeSpecName "kube-api-access-mhlq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:25:08.625044 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.624995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2aaba14b-4684-4767-a2f6-1033f8d533cd" (UID: "2aaba14b-4684-4767-a2f6-1033f8d533cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:25:08.705914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.705810    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aaba14b-4684-4767-a2f6-1033f8d533cd-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:08.705914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:08.705855    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlq6\" (UniqueName: \"kubernetes.io/projected/2aaba14b-4684-4767-a2f6-1033f8d533cd-kube-api-access-mhlq6\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:09.389350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:09.389317    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-jj7lb"
Sep 18 19:25:09.389519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:09.389317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-jj7lb" event={"ID":"2aaba14b-4684-4767-a2f6-1033f8d533cd","Type":"ContainerDied","Data":"fb0e02485d530f558ff60ad05bfbe5c07209fb00c26f3ccee279e75bb83706cc"}
Sep 18 19:25:09.389519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:09.389408    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0e02485d530f558ff60ad05bfbe5c07209fb00c26f3ccee279e75bb83706cc"
Sep 18 19:25:12.029015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.028974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-gnplz"]
Sep 18 19:25:12.029531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.029305    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerName="s3-tls-init-custom"
Sep 18 19:25:12.029531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.029318    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerName="s3-tls-init-custom"
Sep 18 19:25:12.029531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.029337    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerName="download-hf-model"
Sep 18 19:25:12.029531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.029343    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerName="download-hf-model"
Sep 18 19:25:12.029531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.029422    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aaba14b-4684-4767-a2f6-1033f8d533cd" containerName="s3-tls-init-custom"
Sep 18 19:25:12.031666 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.031649    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.034358 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.034333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 18 19:25:12.034518 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.034403    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:25:12.034584 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.034522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:25:12.034584 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.034532    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 19:25:12.035474 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.035458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:25:12.040101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.040078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-gnplz"]
Sep 18 19:25:12.134716 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.134675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.134922 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.134737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbp7\" (UniqueName: \"kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.235611 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.235566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbp7\" (UniqueName: \"kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.235783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.235682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.236101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.236083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.245273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.245251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbp7\" (UniqueName: \"kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7\") pod \"s3-tls-init-serving-gnplz\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") " pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.358131 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.358046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:12.660823 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:12.660739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-gnplz"]
Sep 18 19:25:12.664185 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:25:12.664159    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod842d9ef7_c2aa_4e79_92ee_5f04f21aa02c.slice/crio-f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396 WatchSource:0}: Error finding container f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396: Status 404 returned error can't find the container with id f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396
Sep 18 19:25:13.407692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:13.407659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerStarted","Data":"8526a2baf9c83735bdfd5561db2292089076f206837f20f10e5e73f4a190c8ae"}
Sep 18 19:25:13.407692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:13.407698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerStarted","Data":"f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396"}
Sep 18 19:25:15.418488 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:15.418458    2583 generic.go:356] "Generic (PLEG): container finished" podID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerID="8526a2baf9c83735bdfd5561db2292089076f206837f20f10e5e73f4a190c8ae" exitCode=0
Sep 18 19:25:15.418884 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:15.418528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerDied","Data":"8526a2baf9c83735bdfd5561db2292089076f206837f20f10e5e73f4a190c8ae"}
Sep 18 19:25:16.424247 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:16.424201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerStarted","Data":"63dc9661d877737f7c0a7070cf7b88b9f600ade64ed39f557ef3004ed9007ad5"}
Sep 18 19:25:22.452435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:22.452398    2583 generic.go:356] "Generic (PLEG): container finished" podID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerID="63dc9661d877737f7c0a7070cf7b88b9f600ade64ed39f557ef3004ed9007ad5" exitCode=0
Sep 18 19:25:22.452886 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:22.452480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerDied","Data":"63dc9661d877737f7c0a7070cf7b88b9f600ade64ed39f557ef3004ed9007ad5"}
Sep 18 19:25:23.598969 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.598942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:23.639245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.639216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache\") pod \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") "
Sep 18 19:25:23.639437 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.639251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfbp7\" (UniqueName: \"kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7\") pod \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\" (UID: \"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c\") "
Sep 18 19:25:23.641350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.641326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7" (OuterVolumeSpecName: "kube-api-access-pfbp7") pod "842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" (UID: "842d9ef7-c2aa-4e79-92ee-5f04f21aa02c"). InnerVolumeSpecName "kube-api-access-pfbp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:25:23.716599 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.716505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache" (OuterVolumeSpecName: "model-cache") pod "842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" (UID: "842d9ef7-c2aa-4e79-92ee-5f04f21aa02c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:25:23.740132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.740103    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-model-cache\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:23.740132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:23.740130    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfbp7\" (UniqueName: \"kubernetes.io/projected/842d9ef7-c2aa-4e79-92ee-5f04f21aa02c-kube-api-access-pfbp7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:24.462507 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:24.462480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-gnplz"
Sep 18 19:25:24.462692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:24.462473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-gnplz" event={"ID":"842d9ef7-c2aa-4e79-92ee-5f04f21aa02c","Type":"ContainerDied","Data":"f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396"}
Sep 18 19:25:24.462692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:24.462590    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f594e39a06bc388a8eed41e4b26ee34c07e753fb8a3d45ab341c7ea3222a1396"
Sep 18 19:25:37.570082 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570045    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:37.570522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570354    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerName="s3-tls-init-serving"
Sep 18 19:25:37.570522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570378    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerName="s3-tls-init-serving"
Sep 18 19:25:37.570522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570401    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerName="download-hf-model"
Sep 18 19:25:37.570522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570408    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerName="download-hf-model"
Sep 18 19:25:37.570522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.570476    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="842d9ef7-c2aa-4e79-92ee-5f04f21aa02c" containerName="s3-tls-init-serving"
Sep 18 19:25:37.574520 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.574490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.579330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.579304    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:25:37.579330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.579319    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:25:37.579557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.579391    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9735d5\""
Sep 18 19:25:37.579557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.579441    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:25:37.579657 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.579615    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9735d5-kube-rbac-proxy-sar-config\""
Sep 18 19:25:37.580511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.580485    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9735d5-predictor-serving-cert\""
Sep 18 19:25:37.580626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.580536    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9735d5-dockercfg-9vhqq\""
Sep 18 19:25:37.587173 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.587149    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:37.647942 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.647904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.647942 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.647954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.648164 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.648065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgkn\" (UniqueName: \"kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.648164 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.648106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.648164 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.648132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.748965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.748929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgkn\" (UniqueName: \"kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749171 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.748979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749171 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749171 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749171 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749791 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.749891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.749872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.751482 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.751464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.759025 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.759002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgkn\" (UniqueName: \"kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn\") pod \"isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") " pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:37.885520 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:37.885426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:38.162656 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:38.162627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:38.165242 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:25:38.165214    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54250f6_fc33_49a0_a396_f1345f03f120.slice/crio-5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd WatchSource:0}: Error finding container 5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd: Status 404 returned error can't find the container with id 5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd
Sep 18 19:25:38.526288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:38.526246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerStarted","Data":"2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"}
Sep 18 19:25:38.526488 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:38.526335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerStarted","Data":"5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd"}
Sep 18 19:25:42.544143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:42.544110    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/0.log"
Sep 18 19:25:42.544519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:42.544152    2583 generic.go:356] "Generic (PLEG): container finished" podID="a54250f6-fc33-49a0-a396-f1345f03f120" containerID="2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651" exitCode=1
Sep 18 19:25:42.544519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:42.544233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerDied","Data":"2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"}
Sep 18 19:25:43.549821 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:43.549794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/0.log"
Sep 18 19:25:43.550276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:43.549876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerStarted","Data":"5db3b51251ec0a9a0b0abf68257a8da662652a0730f920d129407211d1508995"}
Sep 18 19:25:44.555770 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.555736    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/1.log"
Sep 18 19:25:44.556174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.556156    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/0.log"
Sep 18 19:25:44.556220 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.556191    2583 generic.go:356] "Generic (PLEG): container finished" podID="a54250f6-fc33-49a0-a396-f1345f03f120" containerID="5db3b51251ec0a9a0b0abf68257a8da662652a0730f920d129407211d1508995" exitCode=1
Sep 18 19:25:44.556258 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.556225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerDied","Data":"5db3b51251ec0a9a0b0abf68257a8da662652a0730f920d129407211d1508995"}
Sep 18 19:25:44.556316 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.556281    2583 scope.go:117] "RemoveContainer" containerID="2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"
Sep 18 19:25:44.556652 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.556635    2583 scope.go:117] "RemoveContainer" containerID="2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"
Sep 18 19:25:44.567177 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:25:44.567153    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_kserve-ci-e2e-test(a54250f6-fc33-49a0-a396-f1345f03f120)\"" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" podUID="a54250f6-fc33-49a0-a396-f1345f03f120"
Sep 18 19:25:44.567415 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:25:44.567390    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_kserve-ci-e2e-test_a54250f6-fc33-49a0-a396-f1345f03f120_0 in pod sandbox 5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd from index: no such id: '2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651'" containerID="2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"
Sep 18 19:25:44.567467 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:44.567427    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_kserve-ci-e2e-test_a54250f6-fc33-49a0-a396-f1345f03f120_0 in pod sandbox 5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd from index: no such id: '2aaec6cacbc3bcca66fdc3247d737cb5e10ee5fc677b4536ee00e61bf9dd6651'"
Sep 18 19:25:45.562399 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:45.562355    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/1.log"
Sep 18 19:25:47.387820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.387783    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:47.512600 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.512570    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/1.log"
Sep 18 19:25:47.514965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.514945    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:47.571636 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.571605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4_a54250f6-fc33-49a0-a396-f1345f03f120/storage-initializer/1.log"
Sep 18 19:25:47.571831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.571707    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"
Sep 18 19:25:47.571831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.571724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4" event={"ID":"a54250f6-fc33-49a0-a396-f1345f03f120","Type":"ContainerDied","Data":"5d0ab1d2c930f2f968fb50b0c3d682127a5e8be9b81e0f3e66057944ffe883dd"}
Sep 18 19:25:47.571831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.571801    2583 scope.go:117] "RemoveContainer" containerID="5db3b51251ec0a9a0b0abf68257a8da662652a0730f920d129407211d1508995"
Sep 18 19:25:47.638517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls\") pod \"a54250f6-fc33-49a0-a396-f1345f03f120\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") "
Sep 18 19:25:47.638517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfgkn\" (UniqueName: \"kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn\") pod \"a54250f6-fc33-49a0-a396-f1345f03f120\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") "
Sep 18 19:25:47.638745 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638571    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location\") pod \"a54250f6-fc33-49a0-a396-f1345f03f120\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") "
Sep 18 19:25:47.638745 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638602    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config\") pod \"a54250f6-fc33-49a0-a396-f1345f03f120\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") "
Sep 18 19:25:47.638745 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert\") pod \"a54250f6-fc33-49a0-a396-f1345f03f120\" (UID: \"a54250f6-fc33-49a0-a396-f1345f03f120\") "
Sep 18 19:25:47.638905 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a54250f6-fc33-49a0-a396-f1345f03f120" (UID: "a54250f6-fc33-49a0-a396-f1345f03f120"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:25:47.639028 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.638994    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a54250f6-fc33-49a0-a396-f1345f03f120" (UID: "a54250f6-fc33-49a0-a396-f1345f03f120"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:25:47.639028 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.639007    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a54250f6-fc33-49a0-a396-f1345f03f120" (UID: "a54250f6-fc33-49a0-a396-f1345f03f120"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:25:47.640577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.640554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a54250f6-fc33-49a0-a396-f1345f03f120" (UID: "a54250f6-fc33-49a0-a396-f1345f03f120"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:25:47.640724 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.640701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn" (OuterVolumeSpecName: "kube-api-access-vfgkn") pod "a54250f6-fc33-49a0-a396-f1345f03f120" (UID: "a54250f6-fc33-49a0-a396-f1345f03f120"). InnerVolumeSpecName "kube-api-access-vfgkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:25:47.739896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.739846    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54250f6-fc33-49a0-a396-f1345f03f120-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:47.739896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.739882    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:47.739896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.739892    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54250f6-fc33-49a0-a396-f1345f03f120-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:47.739896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.739902    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54250f6-fc33-49a0-a396-f1345f03f120-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:47.740161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.739910    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgkn\" (UniqueName: \"kubernetes.io/projected/a54250f6-fc33-49a0-a396-f1345f03f120-kube-api-access-vfgkn\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:25:47.904309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.904220    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:47.907923 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:47.907894    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9735d5-predictor-cd8fc98b4-9f9j4"]
Sep 18 19:25:49.791151 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:49.791108    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" path="/var/lib/kubelet/pods/a54250f6-fc33-49a0-a396-f1345f03f120/volumes"
Sep 18 19:25:52.575479 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575444    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:25:52.575851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575762    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.575851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575775    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.575851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575846    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.575952 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.575952 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575920    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.576012 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.575982    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54250f6-fc33-49a0-a396-f1345f03f120" containerName="storage-initializer"
Sep 18 19:25:52.580318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.580301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.583033 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 18 19:25:52.583204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583185    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:25:52.583340 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:25:52.583340 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:25:52.583340 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583308    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:25:52.583538 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.583432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:25:52.584241 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.584227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 18 19:25:52.590906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.590883    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:25:52.681653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.681618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.681653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.681668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzd98\" (UniqueName: \"kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.681918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.681689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.681918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.681722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.681918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.681812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783129 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783295 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd98\" (UniqueName: \"kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783295 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783295 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783295 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.783949 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.784001 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.783957    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.785778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.785760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.792512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.792485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzd98\" (UniqueName: \"kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98\") pod \"isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:52.891160 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:52.891071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:25:53.164184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:53.164035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:25:53.166897 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:25:53.166862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a14fdfc_54aa_41d7_b834_4d91f5d104a1.slice/crio-42af9d1c82900d0dc10a0a0456ad828ba3acd29fba6693f9224ca2bc66081890 WatchSource:0}: Error finding container 42af9d1c82900d0dc10a0a0456ad828ba3acd29fba6693f9224ca2bc66081890: Status 404 returned error can't find the container with id 42af9d1c82900d0dc10a0a0456ad828ba3acd29fba6693f9224ca2bc66081890
Sep 18 19:25:53.599461 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:53.599422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerStarted","Data":"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"}
Sep 18 19:25:53.599855 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:53.599485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerStarted","Data":"42af9d1c82900d0dc10a0a0456ad828ba3acd29fba6693f9224ca2bc66081890"}
Sep 18 19:25:57.617830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:57.617794    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerID="f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab" exitCode=0
Sep 18 19:25:57.618222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:25:57.617864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerDied","Data":"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"}
Sep 18 19:26:20.745050 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:20.744958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerStarted","Data":"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"}
Sep 18 19:26:23.760810 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:23.760773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerStarted","Data":"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"}
Sep 18 19:26:23.761230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:23.761038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:26:23.761230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:23.761100    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:26:23.762385 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:23.762343    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:26:23.800203 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:23.800152    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podStartSLOduration=3.267671304 podStartE2EDuration="31.800135263s" totalImagesPullingTime="24.93261341s" totalInitContainerRuntime="3.599850549s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:25:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:26:23.79939209 +0000 UTC m=+716.174818382" watchObservedRunningTime="2026-09-18 19:26:23.800135263 +0000 UTC m=+716.175561583"
Sep 18 19:26:24.766871 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:24.766826    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:26:29.771517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:29.771492    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:26:29.771985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:29.771961    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:26:39.772865 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:39.772822    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:26:49.772183 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:49.772140    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:26:59.772696 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:26:59.772647    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:27:09.772030 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:09.771981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:27:19.772256 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:19.772214    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:27:29.772561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:29.772520    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 19:27:39.772545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:39.772507    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:27:43.127681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:43.127608    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:27:43.128083 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:43.127830    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container" containerID="cri-o://3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b" gracePeriod=30
Sep 18 19:27:43.128083 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:43.127927    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kube-rbac-proxy" containerID="cri-o://2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531" gracePeriod=30
Sep 18 19:27:44.108552 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:44.108515    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerID="2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531" exitCode=2
Sep 18 19:27:44.108725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:44.108590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerDied","Data":"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"}
Sep 18 19:27:44.767793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:44.767743    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 19:27:48.323545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.323505    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:27:48.325939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.325921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.328881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.328846    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 19:27:48.328881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.328868    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 18 19:27:48.338735 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.338707    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:27:48.394616 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.394576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.394616 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.394628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.394830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.394676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.394830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.394731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxkh\" (UniqueName: \"kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.394830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.394768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.495930 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.495891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxkh\" (UniqueName: \"kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.495930 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.495937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.495984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.496005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.496037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496480 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.496461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.496700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.496795 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.496778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.498772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.498747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.505664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.505637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxkh\" (UniqueName: \"kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh\") pod \"isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.636115 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.636079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:48.671898 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.671873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:27:48.697539 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.697504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert\") pod \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") "
Sep 18 19:27:48.697699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.697582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzd98\" (UniqueName: \"kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98\") pod \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") "
Sep 18 19:27:48.697699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.697630    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config\") pod \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") "
Sep 18 19:27:48.697699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.697687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location\") pod \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") "
Sep 18 19:27:48.697862 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.697727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls\") pod \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\" (UID: \"3a14fdfc-54aa-41d7-b834-4d91f5d104a1\") "
Sep 18 19:27:48.698388 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.698337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a14fdfc-54aa-41d7-b834-4d91f5d104a1" (UID: "3a14fdfc-54aa-41d7-b834-4d91f5d104a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:27:48.698512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.698418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a14fdfc-54aa-41d7-b834-4d91f5d104a1" (UID: "3a14fdfc-54aa-41d7-b834-4d91f5d104a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:27:48.698576 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.698554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a14fdfc-54aa-41d7-b834-4d91f5d104a1" (UID: "3a14fdfc-54aa-41d7-b834-4d91f5d104a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:27:48.701812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.701780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a14fdfc-54aa-41d7-b834-4d91f5d104a1" (UID: "3a14fdfc-54aa-41d7-b834-4d91f5d104a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:27:48.702207 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.702132    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98" (OuterVolumeSpecName: "kube-api-access-fzd98") pod "3a14fdfc-54aa-41d7-b834-4d91f5d104a1" (UID: "3a14fdfc-54aa-41d7-b834-4d91f5d104a1"). InnerVolumeSpecName "kube-api-access-fzd98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:27:48.799030 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.798985    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:27:48.799030 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.799025    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:27:48.799283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.799041    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:27:48.799283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.799053    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzd98\" (UniqueName: \"kubernetes.io/projected/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-api-access-fzd98\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:27:48.799283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.799062    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a14fdfc-54aa-41d7-b834-4d91f5d104a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:27:48.915113 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:48.915088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:27:48.917394 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:27:48.917351    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfb0f04_7197_45a7_835c_e1b6d56f8ede.slice/crio-64fea6e4d57983720f09a91f5051b0a5770d5675efbd3c7d432705e343b467f5 WatchSource:0}: Error finding container 64fea6e4d57983720f09a91f5051b0a5770d5675efbd3c7d432705e343b467f5: Status 404 returned error can't find the container with id 64fea6e4d57983720f09a91f5051b0a5770d5675efbd3c7d432705e343b467f5
Sep 18 19:27:49.131980 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.131946    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerID="3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b" exitCode=0
Sep 18 19:27:49.132174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.132004    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"
Sep 18 19:27:49.132174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.132030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerDied","Data":"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"}
Sep 18 19:27:49.132174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.132097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn" event={"ID":"3a14fdfc-54aa-41d7-b834-4d91f5d104a1","Type":"ContainerDied","Data":"42af9d1c82900d0dc10a0a0456ad828ba3acd29fba6693f9224ca2bc66081890"}
Sep 18 19:27:49.132174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.132137    2583 scope.go:117] "RemoveContainer" containerID="2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"
Sep 18 19:27:49.133953 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.133925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerStarted","Data":"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"}
Sep 18 19:27:49.134059 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.133988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerStarted","Data":"64fea6e4d57983720f09a91f5051b0a5770d5675efbd3c7d432705e343b467f5"}
Sep 18 19:27:49.141344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.141323    2583 scope.go:117] "RemoveContainer" containerID="3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"
Sep 18 19:27:49.149253 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.149230    2583 scope.go:117] "RemoveContainer" containerID="f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"
Sep 18 19:27:49.156640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.156614    2583 scope.go:117] "RemoveContainer" containerID="2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"
Sep 18 19:27:49.156952 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:27:49.156931    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531\": container with ID starting with 2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531 not found: ID does not exist" containerID="2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"
Sep 18 19:27:49.157015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.156961    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531"} err="failed to get container status \"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531\": rpc error: code = NotFound desc = could not find container \"2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531\": container with ID starting with 2a4d842ba2a90b420c0f033c5c782596e9ce24812be58ab4172a9feb14184531 not found: ID does not exist"
Sep 18 19:27:49.157015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.156983    2583 scope.go:117] "RemoveContainer" containerID="3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"
Sep 18 19:27:49.157241 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:27:49.157225    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b\": container with ID starting with 3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b not found: ID does not exist" containerID="3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"
Sep 18 19:27:49.157280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.157246    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b"} err="failed to get container status \"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b\": rpc error: code = NotFound desc = could not find container \"3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b\": container with ID starting with 3c4d63ad7474ef974618310a80c337d8f9f8024f1aa7247441b0b1c75ea41b6b not found: ID does not exist"
Sep 18 19:27:49.157280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.157258    2583 scope.go:117] "RemoveContainer" containerID="f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"
Sep 18 19:27:49.157542 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:27:49.157524    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab\": container with ID starting with f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab not found: ID does not exist" containerID="f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"
Sep 18 19:27:49.157608 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.157551    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab"} err="failed to get container status \"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab\": rpc error: code = NotFound desc = could not find container \"f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab\": container with ID starting with f21662ba701f1223b1de7e8ce3d607fae8a2f835e59258d421eae71747b3b8ab not found: ID does not exist"
Sep 18 19:27:49.176043 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.175971    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:27:49.183681 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.183650    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-6dc55b9cf4-tbrrn"]
Sep 18 19:27:49.793538 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:49.793458    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" path="/var/lib/kubelet/pods/3a14fdfc-54aa-41d7-b834-4d91f5d104a1/volumes"
Sep 18 19:27:53.152420 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:53.152383    2583 generic.go:356] "Generic (PLEG): container finished" podID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerID="95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae" exitCode=0
Sep 18 19:27:53.152834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:53.152433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerDied","Data":"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"}
Sep 18 19:27:54.158775 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.158740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerStarted","Data":"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"}
Sep 18 19:27:54.159193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.158800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerStarted","Data":"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"}
Sep 18 19:27:54.159193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.159081    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:54.159311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.159254    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:27:54.160487 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.160457    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:27:54.190357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:54.190291    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podStartSLOduration=2.485476809 podStartE2EDuration="6.190271523s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704794714s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:27:54.189600215 +0000 UTC m=+806.565026508" watchObservedRunningTime="2026-09-18 19:27:54.190271523 +0000 UTC m=+806.565697817"
Sep 18 19:27:55.163276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:27:55.163232    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:00.168664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:00.168632    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:28:00.169165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:00.169136    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:10.169616 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:10.169575    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:20.169873 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:20.169831    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:30.169808 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:30.169721    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:40.169250 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:40.169206    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:28:50.169866 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:28:50.169820    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:29:00.169341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:00.169298    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:29:10.170084 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:10.170038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:29:12.356984 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.356946    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357313    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container"
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357327    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container"
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357341    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kube-rbac-proxy"
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kube-rbac-proxy"
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357376    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="storage-initializer"
Sep 18 19:29:12.357383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357382    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="storage-initializer"
Sep 18 19:29:12.357592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357425    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kserve-container"
Sep 18 19:29:12.357592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.357436    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a14fdfc-54aa-41d7-b834-4d91f5d104a1" containerName="kube-rbac-proxy"
Sep 18 19:29:12.360464 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.360446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.362983 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.362953    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 18 19:29:12.363117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.362991    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 18 19:29:12.369560 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.369525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:29:12.399938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.399905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.400141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.399958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghnk\" (UniqueName: \"kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.400141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.400017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.501077 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.501043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.501278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.501096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghnk\" (UniqueName: \"kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.501278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.501142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.501278 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:29:12.501198    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 18 19:29:12.501278 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:29:12.501258    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls podName:50eec08a-ed9d-428d-819f-6877f5ae00ed nodeName:}" failed. No retries permitted until 2026-09-18 19:29:13.001237195 +0000 UTC m=+885.376663469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls") pod "message-dumper-predictor-796c8b7645-z8kn6" (UID: "50eec08a-ed9d-428d-819f-6877f5ae00ed") : secret "message-dumper-predictor-serving-cert" not found
Sep 18 19:29:12.501897 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.501872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:12.511072 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:12.511050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghnk\" (UniqueName: \"kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:13.006492 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:13.006449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:13.008990 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:13.008967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") pod \"message-dumper-predictor-796c8b7645-z8kn6\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") " pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:13.270263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:13.270166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:13.551818 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:13.551723    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:29:13.554161 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:29:13.554129    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eec08a_ed9d_428d_819f_6877f5ae00ed.slice/crio-c96cbb0e8d1a4962114fd84d038aea86a5fad98bf8dc29c88406b51ff2b42872 WatchSource:0}: Error finding container c96cbb0e8d1a4962114fd84d038aea86a5fad98bf8dc29c88406b51ff2b42872: Status 404 returned error can't find the container with id c96cbb0e8d1a4962114fd84d038aea86a5fad98bf8dc29c88406b51ff2b42872
Sep 18 19:29:13.556100 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:13.556084    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:29:14.499532 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:14.499492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerStarted","Data":"c96cbb0e8d1a4962114fd84d038aea86a5fad98bf8dc29c88406b51ff2b42872"}
Sep 18 19:29:15.505877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:15.505845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerStarted","Data":"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"}
Sep 18 19:29:15.506311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:15.505899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerStarted","Data":"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"}
Sep 18 19:29:15.506311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:15.505971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:15.534966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:15.534923    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" podStartSLOduration=2.380731219 podStartE2EDuration="3.534909533s" totalImagesPullingTime="1.154178314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:15.531666567 +0000 UTC m=+887.907092860" watchObservedRunningTime="2026-09-18 19:29:15.534909533 +0000 UTC m=+887.910335826"
Sep 18 19:29:16.510374 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:16.510328    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:16.512200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:16.512177    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:20.169526 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:20.169498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:29:23.523711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:23.523682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:29:28.825003 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:28.824964    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:29:28.825510 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:28.825208    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" containerID="cri-o://b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d" gracePeriod=30
Sep 18 19:29:28.825510 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:28.825257    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kube-rbac-proxy" containerID="cri-o://5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61" gracePeriod=30
Sep 18 19:29:29.565495 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:29.565461    2583 generic.go:356] "Generic (PLEG): container finished" podID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerID="5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61" exitCode=2
Sep 18 19:29:29.565661 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:29.565534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerDied","Data":"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"}
Sep 18 19:29:29.737867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:29.737837    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:29:29.742523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:29.742497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:29:30.163805 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:30.163756    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 19:29:30.169143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:30.169109    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 19:29:34.070238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.070202    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:29:34.119786 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.119750    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:29:34.120118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120103    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="storage-initializer"
Sep 18 19:29:34.120118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120120    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="storage-initializer"
Sep 18 19:29:34.120204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120129    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container"
Sep 18 19:29:34.120204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120135    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container"
Sep 18 19:29:34.120204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120147    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kube-rbac-proxy"
Sep 18 19:29:34.120204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120153    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kube-rbac-proxy"
Sep 18 19:29:34.120320 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120216    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kube-rbac-proxy"
Sep 18 19:29:34.120320 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.120227    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerName="kserve-container"
Sep 18 19:29:34.123413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.123396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.127594 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.127570    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 18 19:29:34.127901 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.127883    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 18 19:29:34.143582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.143546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:29:34.185237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config\") pod \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") "
Sep 18 19:29:34.185237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185131    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzxkh\" (UniqueName: \"kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh\") pod \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") "
Sep 18 19:29:34.185237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185160    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert\") pod \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") "
Sep 18 19:29:34.185237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185206    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location\") pod \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") "
Sep 18 19:29:34.185662 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185425    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls\") pod \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\" (UID: \"6bfb0f04-7197-45a7-835c-e1b6d56f8ede\") "
Sep 18 19:29:34.185720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bfb0f04-7197-45a7-835c-e1b6d56f8ede" (UID: "6bfb0f04-7197-45a7-835c-e1b6d56f8ede"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:29:34.185773 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6bfb0f04-7197-45a7-835c-e1b6d56f8ede" (UID: "6bfb0f04-7197-45a7-835c-e1b6d56f8ede"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:29:34.185843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.185824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6bfb0f04-7197-45a7-835c-e1b6d56f8ede" (UID: "6bfb0f04-7197-45a7-835c-e1b6d56f8ede"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:29:34.187956 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.187926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bfb0f04-7197-45a7-835c-e1b6d56f8ede" (UID: "6bfb0f04-7197-45a7-835c-e1b6d56f8ede"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:29:34.188047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.187932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh" (OuterVolumeSpecName: "kube-api-access-hzxkh") pod "6bfb0f04-7197-45a7-835c-e1b6d56f8ede" (UID: "6bfb0f04-7197-45a7-835c-e1b6d56f8ede"). InnerVolumeSpecName "kube-api-access-hzxkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:29:34.286348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294dp\" (UniqueName: \"kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.286547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.286547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.286547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.286547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.286685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286570    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:29:34.286685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286594    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:29:34.286685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286603    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:29:34.286685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286613    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxkh\" (UniqueName: \"kubernetes.io/projected/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-kube-api-access-hzxkh\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:29:34.286685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.286622    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bfb0f04-7197-45a7-835c-e1b6d56f8ede-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:29:34.387169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.387169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.387465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.387465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.387465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294dp\" (UniqueName: \"kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.387727 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.387704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.388120 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.388096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.388159 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.388146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.389790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.389765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.401485 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.401455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-294dp\" (UniqueName: \"kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp\") pod \"isvc-predictive-lightgbm-predictor-76f4864458-8vtsc\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.433580 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.433536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:34.587913 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.587876    2583 generic.go:356] "Generic (PLEG): container finished" podID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" containerID="b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d" exitCode=0
Sep 18 19:29:34.588102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.587941    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"
Sep 18 19:29:34.588102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.587940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerDied","Data":"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"}
Sep 18 19:29:34.588102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.588016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr" event={"ID":"6bfb0f04-7197-45a7-835c-e1b6d56f8ede","Type":"ContainerDied","Data":"64fea6e4d57983720f09a91f5051b0a5770d5675efbd3c7d432705e343b467f5"}
Sep 18 19:29:34.588102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.588084    2583 scope.go:117] "RemoveContainer" containerID="5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"
Sep 18 19:29:34.597151 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.597129    2583 scope.go:117] "RemoveContainer" containerID="b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"
Sep 18 19:29:34.605223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.605203    2583 scope.go:117] "RemoveContainer" containerID="95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"
Sep 18 19:29:34.612330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.612307    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:29:34.613279 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.613261    2583 scope.go:117] "RemoveContainer" containerID="5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"
Sep 18 19:29:34.613597 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:29:34.613577    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61\": container with ID starting with 5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61 not found: ID does not exist" containerID="5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"
Sep 18 19:29:34.613659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.613607    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61"} err="failed to get container status \"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61\": rpc error: code = NotFound desc = could not find container \"5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61\": container with ID starting with 5b349f2a704a207e0f5c70c00442a0d2eb80862aee98f0566c478cde2e715a61 not found: ID does not exist"
Sep 18 19:29:34.613659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.613636    2583 scope.go:117] "RemoveContainer" containerID="b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"
Sep 18 19:29:34.613919 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:29:34.613899    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d\": container with ID starting with b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d not found: ID does not exist" containerID="b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"
Sep 18 19:29:34.613961 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.613925    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d"} err="failed to get container status \"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d\": rpc error: code = NotFound desc = could not find container \"b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d\": container with ID starting with b0da9e037bdd35195c487cc54254cdf757446ed73017fe6adb01ed86c7935a9d not found: ID does not exist"
Sep 18 19:29:34.613961 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.613942    2583 scope.go:117] "RemoveContainer" containerID="95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"
Sep 18 19:29:34.614201 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:29:34.614184    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae\": container with ID starting with 95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae not found: ID does not exist" containerID="95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"
Sep 18 19:29:34.614257 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.614203    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae"} err="failed to get container status \"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae\": rpc error: code = NotFound desc = could not find container \"95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae\": container with ID starting with 95cb4806be9e5c2e4fa3dcaff88f3fc20c9faf7df4f2af52dc9d73eab1d504ae not found: ID does not exist"
Sep 18 19:29:34.620698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.620677    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-6f947fc4c7-wm8tr"]
Sep 18 19:29:34.771837 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:34.771805    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:29:34.775164 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:29:34.775136    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c73e20_34b3_4bb7_b112_c17629fad1d3.slice/crio-a4204d3bdd2754004150079ff8c43da51cadf710c9e06cc47d6e5cd313ea7a2d WatchSource:0}: Error finding container a4204d3bdd2754004150079ff8c43da51cadf710c9e06cc47d6e5cd313ea7a2d: Status 404 returned error can't find the container with id a4204d3bdd2754004150079ff8c43da51cadf710c9e06cc47d6e5cd313ea7a2d
Sep 18 19:29:35.595791 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:35.595753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerStarted","Data":"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"}
Sep 18 19:29:35.596229 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:35.595811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerStarted","Data":"a4204d3bdd2754004150079ff8c43da51cadf710c9e06cc47d6e5cd313ea7a2d"}
Sep 18 19:29:35.792236 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:35.792200    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bfb0f04-7197-45a7-835c-e1b6d56f8ede" path="/var/lib/kubelet/pods/6bfb0f04-7197-45a7-835c-e1b6d56f8ede/volumes"
Sep 18 19:29:39.614062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:39.614026    2583 generic.go:356] "Generic (PLEG): container finished" podID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerID="184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2" exitCode=0
Sep 18 19:29:39.614463 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:39.614095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerDied","Data":"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"}
Sep 18 19:29:40.620878 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.620845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerStarted","Data":"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"}
Sep 18 19:29:40.621273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.620907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerStarted","Data":"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"}
Sep 18 19:29:40.621273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.621163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:40.621402 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.621306    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:40.622703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.622676    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:29:40.644274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:40.644222    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podStartSLOduration=2.703133648 podStartE2EDuration="6.644202581s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941068933s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:29:40.643122549 +0000 UTC m=+913.018548843" watchObservedRunningTime="2026-09-18 19:29:40.644202581 +0000 UTC m=+913.019628947"
Sep 18 19:29:41.625217 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:41.625178    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:29:46.630081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:46.630046    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:29:46.630586 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:46.630561    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:29:56.630643 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:29:56.630601    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:06.631051 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:06.631006    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:16.631258 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:16.631212    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:26.631306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:26.631263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:36.631203 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:36.631159    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:46.631520 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:46.631436    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:30:50.787221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:30:50.787171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:31:00.787529 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:00.787486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:31:04.671415 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:04.671356    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:31:04.671891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:04.671641    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container" containerID="cri-o://cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e" gracePeriod=30
Sep 18 19:31:04.671891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:04.671726    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kube-rbac-proxy" containerID="cri-o://e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56" gracePeriod=30
Sep 18 19:31:04.987473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:04.987385    2583 generic.go:356] "Generic (PLEG): container finished" podID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerID="e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56" exitCode=2
Sep 18 19:31:04.987473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:04.987457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerDied","Data":"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"}
Sep 18 19:31:06.626165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:06.626117    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:31:07.911901 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:07.911866    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-796c8b7645-z8kn6_50eec08a-ed9d-428d-819f-6877f5ae00ed/kserve-container/0.log"
Sep 18 19:31:08.212641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.212551    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:31:08.212847 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.212801    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kserve-container" containerID="cri-o://850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf" gracePeriod=30
Sep 18 19:31:08.212895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.212847    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kube-rbac-proxy" containerID="cri-o://29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd" gracePeriod=30
Sep 18 19:31:08.454508 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.454480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:31:08.593172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.593130    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") pod \"50eec08a-ed9d-428d-819f-6877f5ae00ed\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") "
Sep 18 19:31:08.593396 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.593224    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ghnk\" (UniqueName: \"kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk\") pod \"50eec08a-ed9d-428d-819f-6877f5ae00ed\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") "
Sep 18 19:31:08.593396 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.593258    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config\") pod \"50eec08a-ed9d-428d-819f-6877f5ae00ed\" (UID: \"50eec08a-ed9d-428d-819f-6877f5ae00ed\") "
Sep 18 19:31:08.593696 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.593667    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50eec08a-ed9d-428d-819f-6877f5ae00ed" (UID: "50eec08a-ed9d-428d-819f-6877f5ae00ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:08.595269 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.595234    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50eec08a-ed9d-428d-819f-6877f5ae00ed" (UID: "50eec08a-ed9d-428d-819f-6877f5ae00ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:08.595269 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.595260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk" (OuterVolumeSpecName: "kube-api-access-9ghnk") pod "50eec08a-ed9d-428d-819f-6877f5ae00ed" (UID: "50eec08a-ed9d-428d-819f-6877f5ae00ed"). InnerVolumeSpecName "kube-api-access-9ghnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:08.694172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.694123    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.694172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.694163    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50eec08a-ed9d-428d-819f-6877f5ae00ed-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:08.694172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:08.694173    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghnk\" (UniqueName: \"kubernetes.io/projected/50eec08a-ed9d-428d-819f-6877f5ae00ed-kube-api-access-9ghnk\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:09.006291 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006251    2583 generic.go:356] "Generic (PLEG): container finished" podID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerID="29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd" exitCode=2
Sep 18 19:31:09.006291 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006283    2583 generic.go:356] "Generic (PLEG): container finished" podID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerID="850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf" exitCode=2
Sep 18 19:31:09.006750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006320    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"
Sep 18 19:31:09.006750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerDied","Data":"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"}
Sep 18 19:31:09.006750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerDied","Data":"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"}
Sep 18 19:31:09.006750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6" event={"ID":"50eec08a-ed9d-428d-819f-6877f5ae00ed","Type":"ContainerDied","Data":"c96cbb0e8d1a4962114fd84d038aea86a5fad98bf8dc29c88406b51ff2b42872"}
Sep 18 19:31:09.006750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.006482    2583 scope.go:117] "RemoveContainer" containerID="29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"
Sep 18 19:31:09.015623 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.015598    2583 scope.go:117] "RemoveContainer" containerID="850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"
Sep 18 19:31:09.023849 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.023826    2583 scope.go:117] "RemoveContainer" containerID="29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"
Sep 18 19:31:09.024160 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:31:09.024136    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd\": container with ID starting with 29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd not found: ID does not exist" containerID="29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"
Sep 18 19:31:09.024237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024174    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"} err="failed to get container status \"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd\": rpc error: code = NotFound desc = could not find container \"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd\": container with ID starting with 29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd not found: ID does not exist"
Sep 18 19:31:09.024237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024198    2583 scope.go:117] "RemoveContainer" containerID="850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"
Sep 18 19:31:09.024501 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:31:09.024484    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf\": container with ID starting with 850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf not found: ID does not exist" containerID="850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"
Sep 18 19:31:09.024545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024508    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"} err="failed to get container status \"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf\": rpc error: code = NotFound desc = could not find container \"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf\": container with ID starting with 850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf not found: ID does not exist"
Sep 18 19:31:09.024545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024523    2583 scope.go:117] "RemoveContainer" containerID="29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"
Sep 18 19:31:09.024747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024724    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd"} err="failed to get container status \"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd\": rpc error: code = NotFound desc = could not find container \"29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd\": container with ID starting with 29f68d6f7fe735c23fdc036e8b5f851f2898878df24b08a21041965f249af4dd not found: ID does not exist"
Sep 18 19:31:09.024812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024747    2583 scope.go:117] "RemoveContainer" containerID="850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"
Sep 18 19:31:09.024969 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.024951    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf"} err="failed to get container status \"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf\": rpc error: code = NotFound desc = could not find container \"850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf\": container with ID starting with 850a6ce714e3ad53eca34c911c5dcac7c174d00c14d1d525237a7c646efa2adf not found: ID does not exist"
Sep 18 19:31:09.033942 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.033900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:31:09.035699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.035678    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-796c8b7645-z8kn6"]
Sep 18 19:31:09.791090 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.791043    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" path="/var/lib/kubelet/pods/50eec08a-ed9d-428d-819f-6877f5ae00ed/volumes"
Sep 18 19:31:09.826635 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.826598    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:31:09.827005 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.826986    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kserve-container"
Sep 18 19:31:09.827089 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.827007    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kserve-container"
Sep 18 19:31:09.827089 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.827033    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kube-rbac-proxy"
Sep 18 19:31:09.827089 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.827044    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kube-rbac-proxy"
Sep 18 19:31:09.827239 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.827136    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kserve-container"
Sep 18 19:31:09.827239 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.827149    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eec08a-ed9d-428d-819f-6877f5ae00ed" containerName="kube-rbac-proxy"
Sep 18 19:31:09.831761 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.831735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:09.834511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.834488    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:31:09.834655 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.834557    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 18 19:31:09.843810 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.843785    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:31:09.905438 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.905390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rt7\" (UniqueName: \"kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:09.905438 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.905437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:09.905705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.905528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:09.905705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.905564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:09.905705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:09.905600    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.006873 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.006839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rt7\" (UniqueName: \"kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.006873 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.006879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.006927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.006950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.006992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007548 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.007509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007858 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.007836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.007966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.007946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.009378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.009341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.016850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.016825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rt7\" (UniqueName: \"kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7\") pod \"isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.142764 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.142720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:10.416997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.416968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:31:10.419603 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:31:10.419572    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6679703_b1d3_4ff8_81aa_a93eb6592393.slice/crio-eb50063218721afbbc2b6df48e3589c93c42ef5ff232d1fc89756b1da3effc28 WatchSource:0}: Error finding container eb50063218721afbbc2b6df48e3589c93c42ef5ff232d1fc89756b1da3effc28: Status 404 returned error can't find the container with id eb50063218721afbbc2b6df48e3589c93c42ef5ff232d1fc89756b1da3effc28
Sep 18 19:31:10.512336 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.512303    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:31:10.611591 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.611553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls\") pod \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") "
Sep 18 19:31:10.611764 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.611661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-294dp\" (UniqueName: \"kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp\") pod \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") "
Sep 18 19:31:10.611764 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.611709    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config\") pod \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") "
Sep 18 19:31:10.611764 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.611752    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert\") pod \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") "
Sep 18 19:31:10.611906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.611794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location\") pod \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\" (UID: \"e8c73e20-34b3-4bb7-b112-c17629fad1d3\") "
Sep 18 19:31:10.612168 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.612135    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8c73e20-34b3-4bb7-b112-c17629fad1d3" (UID: "e8c73e20-34b3-4bb7-b112-c17629fad1d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:10.612314 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.612139    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8c73e20-34b3-4bb7-b112-c17629fad1d3" (UID: "e8c73e20-34b3-4bb7-b112-c17629fad1d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:31:10.612314 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.612193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8c73e20-34b3-4bb7-b112-c17629fad1d3" (UID: "e8c73e20-34b3-4bb7-b112-c17629fad1d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:31:10.613767 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.613746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8c73e20-34b3-4bb7-b112-c17629fad1d3" (UID: "e8c73e20-34b3-4bb7-b112-c17629fad1d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:31:10.613882 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.613798    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp" (OuterVolumeSpecName: "kube-api-access-294dp") pod "e8c73e20-34b3-4bb7-b112-c17629fad1d3" (UID: "e8c73e20-34b3-4bb7-b112-c17629fad1d3"). InnerVolumeSpecName "kube-api-access-294dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:31:10.712843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.712739    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294dp\" (UniqueName: \"kubernetes.io/projected/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-api-access-294dp\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:10.712843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.712782    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:10.712843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.712801    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c73e20-34b3-4bb7-b112-c17629fad1d3-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:10.712843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.712820    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c73e20-34b3-4bb7-b112-c17629fad1d3-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:10.712843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:10.712837    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c73e20-34b3-4bb7-b112-c17629fad1d3-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:31:11.017843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.017803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerStarted","Data":"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"}
Sep 18 19:31:11.018329 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.017877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerStarted","Data":"eb50063218721afbbc2b6df48e3589c93c42ef5ff232d1fc89756b1da3effc28"}
Sep 18 19:31:11.019959 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.019931    2583 generic.go:356] "Generic (PLEG): container finished" podID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerID="cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e" exitCode=0
Sep 18 19:31:11.020094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.019983    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"
Sep 18 19:31:11.020094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.019984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerDied","Data":"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"}
Sep 18 19:31:11.020094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.020052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc" event={"ID":"e8c73e20-34b3-4bb7-b112-c17629fad1d3","Type":"ContainerDied","Data":"a4204d3bdd2754004150079ff8c43da51cadf710c9e06cc47d6e5cd313ea7a2d"}
Sep 18 19:31:11.020094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.020090    2583 scope.go:117] "RemoveContainer" containerID="e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"
Sep 18 19:31:11.029289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.029265    2583 scope.go:117] "RemoveContainer" containerID="cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"
Sep 18 19:31:11.037838 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.037816    2583 scope.go:117] "RemoveContainer" containerID="184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"
Sep 18 19:31:11.048106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.048071    2583 scope.go:117] "RemoveContainer" containerID="e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"
Sep 18 19:31:11.048470 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:31:11.048449    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56\": container with ID starting with e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56 not found: ID does not exist" containerID="e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"
Sep 18 19:31:11.048587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.048476    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56"} err="failed to get container status \"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56\": rpc error: code = NotFound desc = could not find container \"e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56\": container with ID starting with e2e841e4aa6c22b06d423240e67f3d816b207bf4890b299673f84646e16c0a56 not found: ID does not exist"
Sep 18 19:31:11.048587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.048494    2583 scope.go:117] "RemoveContainer" containerID="cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"
Sep 18 19:31:11.048821 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:31:11.048802    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e\": container with ID starting with cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e not found: ID does not exist" containerID="cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"
Sep 18 19:31:11.048880 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.048826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e"} err="failed to get container status \"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e\": rpc error: code = NotFound desc = could not find container \"cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e\": container with ID starting with cd89f5751db1499689022e453c3697593fe9e2d8ea9aa191060450adee9bc92e not found: ID does not exist"
Sep 18 19:31:11.048880 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.048841    2583 scope.go:117] "RemoveContainer" containerID="184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"
Sep 18 19:31:11.049152 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:31:11.049133    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2\": container with ID starting with 184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2 not found: ID does not exist" containerID="184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"
Sep 18 19:31:11.049225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.049157    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2"} err="failed to get container status \"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2\": rpc error: code = NotFound desc = could not find container \"184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2\": container with ID starting with 184fbadbf41008b2200fcc3df0a1ca6a75a7b0959aa35dcb5a3f4d7cfb0fd3f2 not found: ID does not exist"
Sep 18 19:31:11.063957 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.060972    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:31:11.065627 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.065599    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-76f4864458-8vtsc"]
Sep 18 19:31:11.791630 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:11.791596    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" path="/var/lib/kubelet/pods/e8c73e20-34b3-4bb7-b112-c17629fad1d3/volumes"
Sep 18 19:31:15.039684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:15.039650    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerID="bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd" exitCode=0
Sep 18 19:31:15.040115 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:15.039724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerDied","Data":"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"}
Sep 18 19:31:16.046333 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:16.046300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerStarted","Data":"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"}
Sep 18 19:31:16.046698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:16.046385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerStarted","Data":"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"}
Sep 18 19:31:16.046698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:16.046580    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:16.076221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:16.076163    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podStartSLOduration=3.230078656 podStartE2EDuration="7.076144443s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.846065787s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:31:16.074392752 +0000 UTC m=+1008.449819038" watchObservedRunningTime="2026-09-18 19:31:16.076144443 +0000 UTC m=+1008.451570735"
Sep 18 19:31:17.051048 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:17.051014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:23.060174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:23.060144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:31:53.061276 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:31:53.061233    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:03.060818 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:03.060769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:13.060998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:13.060905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:23.061477 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:23.061434    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:33.061050 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:33.060996    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:43.600463 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:43.600428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:32:49.977441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:49.977328    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:32:49.977834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:49.977578    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" containerID="cri-o://d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a" gracePeriod=30
Sep 18 19:32:49.977834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:49.977667    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kube-rbac-proxy" containerID="cri-o://5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d" gracePeriod=30
Sep 18 19:32:50.457106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:50.457074    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerID="5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d" exitCode=2
Sep 18 19:32:50.457281 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:50.457113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerDied","Data":"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"}
Sep 18 19:32:53.056189 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:53.056145    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 19:32:53.061618 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:53.061579    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:32:55.278182 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.278140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:32:55.278950 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.278927    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container"
Sep 18 19:32:55.278950 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.278953    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container"
Sep 18 19:32:55.279145 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.278966    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kube-rbac-proxy"
Sep 18 19:32:55.279145 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.278975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kube-rbac-proxy"
Sep 18 19:32:55.279145 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.279001    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="storage-initializer"
Sep 18 19:32:55.279145 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.279009    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="storage-initializer"
Sep 18 19:32:55.279341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.279198    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kserve-container"
Sep 18 19:32:55.279341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.279220    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c73e20-34b3-4bb7-b112-c17629fad1d3" containerName="kube-rbac-proxy"
Sep 18 19:32:55.283698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.283345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.288236 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.288208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:32:55.288411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.288298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 18 19:32:55.301379 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.301336    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:32:55.397470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.397394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.397470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.397447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.397695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.397491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.397695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.397551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdrh\" (UniqueName: \"kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.397695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.397604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.425938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.425907    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:32:55.480484 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.480391    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerID="d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a" exitCode=0
Sep 18 19:32:55.480484 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.480436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerDied","Data":"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"}
Sep 18 19:32:55.480484 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.480457    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"
Sep 18 19:32:55.480757 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.480510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k" event={"ID":"d6679703-b1d3-4ff8-81aa-a93eb6592393","Type":"ContainerDied","Data":"eb50063218721afbbc2b6df48e3589c93c42ef5ff232d1fc89756b1da3effc28"}
Sep 18 19:32:55.480757 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.480541    2583 scope.go:117] "RemoveContainer" containerID="5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"
Sep 18 19:32:55.490186 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.490164    2583 scope.go:117] "RemoveContainer" containerID="d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"
Sep 18 19:32:55.498556 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498530    2583 scope.go:117] "RemoveContainer" containerID="bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"
Sep 18 19:32:55.498653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6rt7\" (UniqueName: \"kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7\") pod \"d6679703-b1d3-4ff8-81aa-a93eb6592393\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") "
Sep 18 19:32:55.498723 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location\") pod \"d6679703-b1d3-4ff8-81aa-a93eb6592393\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") "
Sep 18 19:32:55.498723 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config\") pod \"d6679703-b1d3-4ff8-81aa-a93eb6592393\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") "
Sep 18 19:32:55.498837 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls\") pod \"d6679703-b1d3-4ff8-81aa-a93eb6592393\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") "
Sep 18 19:32:55.498837 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert\") pod \"d6679703-b1d3-4ff8-81aa-a93eb6592393\" (UID: \"d6679703-b1d3-4ff8-81aa-a93eb6592393\") "
Sep 18 19:32:55.498945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.498945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.499058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.498983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.499058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdrh\" (UniqueName: \"kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.499155 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499092    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6679703-b1d3-4ff8-81aa-a93eb6592393" (UID: "d6679703-b1d3-4ff8-81aa-a93eb6592393"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:55.499207 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.499708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499266    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:32:55.499708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.499898 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.499873    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6679703-b1d3-4ff8-81aa-a93eb6592393" (UID: "d6679703-b1d3-4ff8-81aa-a93eb6592393"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:32:55.500146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.500092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.500278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.500151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6679703-b1d3-4ff8-81aa-a93eb6592393" (UID: "d6679703-b1d3-4ff8-81aa-a93eb6592393"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:32:55.500278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.500178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.501392 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.501324    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7" (OuterVolumeSpecName: "kube-api-access-d6rt7") pod "d6679703-b1d3-4ff8-81aa-a93eb6592393" (UID: "d6679703-b1d3-4ff8-81aa-a93eb6592393"). InnerVolumeSpecName "kube-api-access-d6rt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:32:55.501596 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.501570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6679703-b1d3-4ff8-81aa-a93eb6592393" (UID: "d6679703-b1d3-4ff8-81aa-a93eb6592393"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:32:55.502407 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.502355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.511053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.511028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdrh\" (UniqueName: \"kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh\") pod \"isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.519476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.519451    2583 scope.go:117] "RemoveContainer" containerID="5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"
Sep 18 19:32:55.519896 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:32:55.519870    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d\": container with ID starting with 5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d not found: ID does not exist" containerID="5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"
Sep 18 19:32:55.519949 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.519908    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d"} err="failed to get container status \"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d\": rpc error: code = NotFound desc = could not find container \"5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d\": container with ID starting with 5f7916ac5656415fc0a714bf58b22f988ca457a01735cb5a718e4e9bfa998f6d not found: ID does not exist"
Sep 18 19:32:55.519949 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.519926    2583 scope.go:117] "RemoveContainer" containerID="d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"
Sep 18 19:32:55.520230 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:32:55.520213    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a\": container with ID starting with d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a not found: ID does not exist" containerID="d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"
Sep 18 19:32:55.520297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.520235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a"} err="failed to get container status \"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a\": rpc error: code = NotFound desc = could not find container \"d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a\": container with ID starting with d7741dd8666df746fb86b1f32e6f66d8eed51273073ea3f80aaa9b058a591a4a not found: ID does not exist"
Sep 18 19:32:55.520297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.520267    2583 scope.go:117] "RemoveContainer" containerID="bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"
Sep 18 19:32:55.520563 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:32:55.520542    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd\": container with ID starting with bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd not found: ID does not exist" containerID="bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"
Sep 18 19:32:55.520626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.520574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd"} err="failed to get container status \"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd\": rpc error: code = NotFound desc = could not find container \"bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd\": container with ID starting with bc891cc2c9b958d8863f327b40212bef3c0fba9725d47deff18e5be4c3a645bd not found: ID does not exist"
Sep 18 19:32:55.594734 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.594698    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:32:55.600221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.600199    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6679703-b1d3-4ff8-81aa-a93eb6592393-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:32:55.600325 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.600222    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6679703-b1d3-4ff8-81aa-a93eb6592393-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:32:55.600325 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.600236    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rt7\" (UniqueName: \"kubernetes.io/projected/d6679703-b1d3-4ff8-81aa-a93eb6592393-kube-api-access-d6rt7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:32:55.600325 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.600249    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6679703-b1d3-4ff8-81aa-a93eb6592393-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:32:55.806710 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.806674    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:32:55.812919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.812891    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-84c6985b7d-4467k"]
Sep 18 19:32:55.876898 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:55.876869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:32:55.879353 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:32:55.879327    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db57f4d_1800_43a7_9189_0786d9a54691.slice/crio-fff18a6dfdda63e21f58dbb8e2d4864216a10e7f723b3c4608fa13400448dbcc WatchSource:0}: Error finding container fff18a6dfdda63e21f58dbb8e2d4864216a10e7f723b3c4608fa13400448dbcc: Status 404 returned error can't find the container with id fff18a6dfdda63e21f58dbb8e2d4864216a10e7f723b3c4608fa13400448dbcc
Sep 18 19:32:56.487938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:56.487898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerStarted","Data":"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"}
Sep 18 19:32:56.488332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:56.487973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerStarted","Data":"fff18a6dfdda63e21f58dbb8e2d4864216a10e7f723b3c4608fa13400448dbcc"}
Sep 18 19:32:57.791848 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:32:57.791802    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" path="/var/lib/kubelet/pods/d6679703-b1d3-4ff8-81aa-a93eb6592393/volumes"
Sep 18 19:33:00.506356 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:00.506318    2583 generic.go:356] "Generic (PLEG): container finished" podID="0db57f4d-1800-43a7-9189-0786d9a54691" containerID="3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32" exitCode=0
Sep 18 19:33:00.506783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:00.506393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerDied","Data":"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"}
Sep 18 19:33:01.513462 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:01.513416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerStarted","Data":"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"}
Sep 18 19:33:01.513908 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:01.513480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerStarted","Data":"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"}
Sep 18 19:33:01.513908 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:01.513688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:33:01.553341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:01.553269    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podStartSLOduration=2.614366521 podStartE2EDuration="6.553249006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.938882485s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:33:01.551244223 +0000 UTC m=+1113.926670516" watchObservedRunningTime="2026-09-18 19:33:01.553249006 +0000 UTC m=+1113.928675300"
Sep 18 19:33:02.517989 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:02.517953    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:33:08.527855 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:08.527822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:33:38.528837 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:38.528798    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:33:48.528577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:48.528486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:33:58.529207 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:33:58.529161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:34:08.529086 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:08.529036    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:34:12.786938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:12.786889    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 19:34:23.302213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:23.302179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:34:25.190869 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:25.190830    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:34:25.191413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:25.191144    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container" containerID="cri-o://c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940" gracePeriod=30
Sep 18 19:34:25.191413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:25.191321    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kube-rbac-proxy" containerID="cri-o://10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b" gracePeriod=30
Sep 18 19:34:25.872872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:25.872835    2583 generic.go:356] "Generic (PLEG): container finished" podID="0db57f4d-1800-43a7-9189-0786d9a54691" containerID="10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b" exitCode=2
Sep 18 19:34:25.873053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:25.872896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerDied","Data":"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"}
Sep 18 19:34:28.522620 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:28.522573    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 19:34:29.768634 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:29.768605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:34:29.770889 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:29.770872    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.417572    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418291    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kube-rbac-proxy"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418312    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kube-rbac-proxy"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418327    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="storage-initializer"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418336    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="storage-initializer"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418394    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418404    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418553    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kserve-container"
Sep 18 19:34:30.420577 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.418563    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6679703-b1d3-4ff8-81aa-a93eb6592393" containerName="kube-rbac-proxy"
Sep 18 19:34:30.423852 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.423831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.426550 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.426533    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 18 19:34:30.427068 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.427051    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:34:30.433793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.433770    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:34:30.514787 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.514754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsm5\" (UniqueName: \"kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.514962 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.514816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.514962 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.514859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.514962 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.514903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.515118 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.514980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.615533 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.615496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.615704 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.615611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsm5\" (UniqueName: \"kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.615704 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.615664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.615704 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.615695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.615888 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.615756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.616188 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.616162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.616274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.616251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.616422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.616383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.618260 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.618243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.624528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.624470    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:34:30.626652 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.626626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsm5\" (UniqueName: \"kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5\") pod \"isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.716909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.716817    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location\") pod \"0db57f4d-1800-43a7-9189-0786d9a54691\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") "
Sep 18 19:34:30.716909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.716878    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert\") pod \"0db57f4d-1800-43a7-9189-0786d9a54691\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") "
Sep 18 19:34:30.717117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.716911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config\") pod \"0db57f4d-1800-43a7-9189-0786d9a54691\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") "
Sep 18 19:34:30.717117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.716983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls\") pod \"0db57f4d-1800-43a7-9189-0786d9a54691\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") "
Sep 18 19:34:30.717117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.717031    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cdrh\" (UniqueName: \"kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh\") pod \"0db57f4d-1800-43a7-9189-0786d9a54691\" (UID: \"0db57f4d-1800-43a7-9189-0786d9a54691\") "
Sep 18 19:34:30.717357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.717201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0db57f4d-1800-43a7-9189-0786d9a54691" (UID: "0db57f4d-1800-43a7-9189-0786d9a54691"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:34:30.717357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.717271    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0db57f4d-1800-43a7-9189-0786d9a54691" (UID: "0db57f4d-1800-43a7-9189-0786d9a54691"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:30.717357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.717291    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0db57f4d-1800-43a7-9189-0786d9a54691-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.717357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.717277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0db57f4d-1800-43a7-9189-0786d9a54691" (UID: "0db57f4d-1800-43a7-9189-0786d9a54691"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:34:30.719112 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.719088    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0db57f4d-1800-43a7-9189-0786d9a54691" (UID: "0db57f4d-1800-43a7-9189-0786d9a54691"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:34:30.719195 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.719089    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh" (OuterVolumeSpecName: "kube-api-access-5cdrh") pod "0db57f4d-1800-43a7-9189-0786d9a54691" (UID: "0db57f4d-1800-43a7-9189-0786d9a54691"). InnerVolumeSpecName "kube-api-access-5cdrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:34:30.784353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.784305    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:30.818578 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.818547    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.818578 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.818575    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0db57f4d-1800-43a7-9189-0786d9a54691-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.818578 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.818584    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cdrh\" (UniqueName: \"kubernetes.io/projected/0db57f4d-1800-43a7-9189-0786d9a54691-kube-api-access-5cdrh\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.818815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.818593    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0db57f4d-1800-43a7-9189-0786d9a54691-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:34:30.896324 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.896280    2583 generic.go:356] "Generic (PLEG): container finished" podID="0db57f4d-1800-43a7-9189-0786d9a54691" containerID="c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940" exitCode=0
Sep 18 19:34:30.896511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.896351    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"
Sep 18 19:34:30.896511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.896352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerDied","Data":"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"}
Sep 18 19:34:30.896511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.896437    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8" event={"ID":"0db57f4d-1800-43a7-9189-0786d9a54691","Type":"ContainerDied","Data":"fff18a6dfdda63e21f58dbb8e2d4864216a10e7f723b3c4608fa13400448dbcc"}
Sep 18 19:34:30.896511 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.896467    2583 scope.go:117] "RemoveContainer" containerID="10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"
Sep 18 19:34:30.905684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.905663    2583 scope.go:117] "RemoveContainer" containerID="c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"
Sep 18 19:34:30.913427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.913402    2583 scope.go:117] "RemoveContainer" containerID="3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"
Sep 18 19:34:30.921452 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.921429    2583 scope.go:117] "RemoveContainer" containerID="10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"
Sep 18 19:34:30.921814 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:34:30.921794    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b\": container with ID starting with 10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b not found: ID does not exist" containerID="10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"
Sep 18 19:34:30.921874 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.921829    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b"} err="failed to get container status \"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b\": rpc error: code = NotFound desc = could not find container \"10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b\": container with ID starting with 10eced7a6d72e23497069a05e4f4eb66be72844db3a4726a3059103895acaf4b not found: ID does not exist"
Sep 18 19:34:30.921874 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.921850    2583 scope.go:117] "RemoveContainer" containerID="c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"
Sep 18 19:34:30.921987 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.921803    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:34:30.922146 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:34:30.922127    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940\": container with ID starting with c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940 not found: ID does not exist" containerID="c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"
Sep 18 19:34:30.922201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.922150    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940"} err="failed to get container status \"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940\": rpc error: code = NotFound desc = could not find container \"c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940\": container with ID starting with c589327024b0442fb6a3fc4f8a2fc22ea13d140301001634c6f73e41a6956940 not found: ID does not exist"
Sep 18 19:34:30.922201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.922165    2583 scope.go:117] "RemoveContainer" containerID="3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"
Sep 18 19:34:30.922455 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:34:30.922435    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32\": container with ID starting with 3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32 not found: ID does not exist" containerID="3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"
Sep 18 19:34:30.922503 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.922463    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32"} err="failed to get container status \"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32\": rpc error: code = NotFound desc = could not find container \"3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32\": container with ID starting with 3ba8f7f72621302b3d8f39c809c8f8e7a238c9e5830bf5aafd8568b032d50c32 not found: ID does not exist"
Sep 18 19:34:30.925830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:30.925806    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-748c464b49-8dhb8"]
Sep 18 19:34:31.060349 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:31.060325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:34:31.062073 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:34:31.062049    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3071ce5_8947_44c8_b67b_f3c2e33a3b9a.slice/crio-b34a2ffaa2d9e6054babe578a358b03e723528ae21aafb97b6601c65ae82b4bc WatchSource:0}: Error finding container b34a2ffaa2d9e6054babe578a358b03e723528ae21aafb97b6601c65ae82b4bc: Status 404 returned error can't find the container with id b34a2ffaa2d9e6054babe578a358b03e723528ae21aafb97b6601c65ae82b4bc
Sep 18 19:34:31.792042 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:31.792012    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" path="/var/lib/kubelet/pods/0db57f4d-1800-43a7-9189-0786d9a54691/volumes"
Sep 18 19:34:31.902699 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:31.902656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerStarted","Data":"711179caa68617938ebd4da50f473cdd49d8434f1ef909015c796efd3309ea45"}
Sep 18 19:34:31.902881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:31.902729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerStarted","Data":"b34a2ffaa2d9e6054babe578a358b03e723528ae21aafb97b6601c65ae82b4bc"}
Sep 18 19:34:34.924490 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:34.924453    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerID="711179caa68617938ebd4da50f473cdd49d8434f1ef909015c796efd3309ea45" exitCode=0
Sep 18 19:34:34.924918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:34.924508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerDied","Data":"711179caa68617938ebd4da50f473cdd49d8434f1ef909015c796efd3309ea45"}
Sep 18 19:34:35.931216 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:35.931181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerStarted","Data":"f6197a76419ab6386cc048ea38a2ec951d6ec61157e1a2fc9b476b8ab8547c8b"}
Sep 18 19:34:35.931646 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:35.931241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerStarted","Data":"afae3128a2a2b3d094810ff3fdf8953d43bd88d3918dfd0fe4132ac0921cd5c6"}
Sep 18 19:34:35.931646 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:35.931434    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:35.931646 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:35.931484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:34:35.960988 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:35.960921    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podStartSLOduration=2.271271843 podStartE2EDuration="5.960903611s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.689631768s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:34:35.958394953 +0000 UTC m=+1208.333821237" watchObservedRunningTime="2026-09-18 19:34:35.960903611 +0000 UTC m=+1208.336329904"
Sep 18 19:34:41.941441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:34:41.941409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:35:11.943278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:35:11.943187    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:35:21.942278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:35:21.942229    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:35:31.942093 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:35:31.942039    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:35:41.943020 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:35:41.942974    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:35:51.942620 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:35:51.942577    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:36:01.303976 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:01.303943    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:36:10.545456 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:10.545424    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:36:10.545827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:10.545678    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" containerID="cri-o://afae3128a2a2b3d094810ff3fdf8953d43bd88d3918dfd0fe4132ac0921cd5c6" gracePeriod=30
Sep 18 19:36:10.545827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:10.545754    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kube-rbac-proxy" containerID="cri-o://f6197a76419ab6386cc048ea38a2ec951d6ec61157e1a2fc9b476b8ab8547c8b" gracePeriod=30
Sep 18 19:36:10.786862 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:10.786823    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 19:36:11.326177 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:11.326140    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerID="f6197a76419ab6386cc048ea38a2ec951d6ec61157e1a2fc9b476b8ab8547c8b" exitCode=2
Sep 18 19:36:11.326353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:11.326212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerDied","Data":"f6197a76419ab6386cc048ea38a2ec951d6ec61157e1a2fc9b476b8ab8547c8b"}
Sep 18 19:36:11.935981 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:11.935936    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 18 19:36:16.349640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.349603    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerID="afae3128a2a2b3d094810ff3fdf8953d43bd88d3918dfd0fe4132ac0921cd5c6" exitCode=0
Sep 18 19:36:16.350027 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.349674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerDied","Data":"afae3128a2a2b3d094810ff3fdf8953d43bd88d3918dfd0fe4132ac0921cd5c6"}
Sep 18 19:36:16.387471 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.387443    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:36:16.429599 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config\") pod \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") "
Sep 18 19:36:16.429792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert\") pod \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") "
Sep 18 19:36:16.429792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location\") pod \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") "
Sep 18 19:36:16.429792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429690    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls\") pod \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") "
Sep 18 19:36:16.429792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429739    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwsm5\" (UniqueName: \"kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5\") pod \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\" (UID: \"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a\") "
Sep 18 19:36:16.430069 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.429972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" (UID: "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:16.430137 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.430064    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" (UID: "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:36:16.430196 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.430156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" (UID: "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:36:16.431778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.431746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" (UID: "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:36:16.431886 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.431867    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5" (OuterVolumeSpecName: "kube-api-access-wwsm5") pod "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" (UID: "e3071ce5-8947-44c8-b67b-f3c2e33a3b9a"). InnerVolumeSpecName "kube-api-access-wwsm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:16.530429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.530302    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.530429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.530332    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.530429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.530343    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.530429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.530353    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsm5\" (UniqueName: \"kubernetes.io/projected/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-api-access-wwsm5\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:16.530429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:16.530395    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:17.356274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.356244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"
Sep 18 19:36:17.356706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.356239    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4" event={"ID":"e3071ce5-8947-44c8-b67b-f3c2e33a3b9a","Type":"ContainerDied","Data":"b34a2ffaa2d9e6054babe578a358b03e723528ae21aafb97b6601c65ae82b4bc"}
Sep 18 19:36:17.356706 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.356409    2583 scope.go:117] "RemoveContainer" containerID="f6197a76419ab6386cc048ea38a2ec951d6ec61157e1a2fc9b476b8ab8547c8b"
Sep 18 19:36:17.365262 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.365239    2583 scope.go:117] "RemoveContainer" containerID="afae3128a2a2b3d094810ff3fdf8953d43bd88d3918dfd0fe4132ac0921cd5c6"
Sep 18 19:36:17.374710 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.374688    2583 scope.go:117] "RemoveContainer" containerID="711179caa68617938ebd4da50f473cdd49d8434f1ef909015c796efd3309ea45"
Sep 18 19:36:17.385853 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.385716    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:36:17.389357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.389330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-54dcd58b96-b9rm4"]
Sep 18 19:36:17.792108 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:17.792075    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" path="/var/lib/kubelet/pods/e3071ce5-8947-44c8-b67b-f3c2e33a3b9a/volumes"
Sep 18 19:36:33.170492 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170453    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-583a616d"]
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170761    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170773    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170785    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="storage-initializer"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="storage-initializer"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170798    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="storage-initializer"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170803    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="storage-initializer"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170812    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170817    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170823    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170828    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170833    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170839    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170894    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kserve-container"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170902    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170910    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db57f4d-1800-43a7-9189-0786d9a54691" containerName="kube-rbac-proxy"
Sep 18 19:36:33.170938 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.170916    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3071ce5-8947-44c8-b67b-f3c2e33a3b9a" containerName="kserve-container"
Sep 18 19:36:33.175204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.175188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:33.178541 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.178519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:36:33.178945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.178929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:36:33.179869 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.179847    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:36:33.182067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.181968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-583a616d"]
Sep 18 19:36:33.269455 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.269424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgqq\" (UniqueName: \"kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq\") pod \"curl-metrics-583a616d\" (UID: \"60a4b7af-d281-4090-9e4d-3dee18428990\") " pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:33.370504 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.370475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgqq\" (UniqueName: \"kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq\") pod \"curl-metrics-583a616d\" (UID: \"60a4b7af-d281-4090-9e4d-3dee18428990\") " pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:33.381322 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.381298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgqq\" (UniqueName: \"kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq\") pod \"curl-metrics-583a616d\" (UID: \"60a4b7af-d281-4090-9e4d-3dee18428990\") " pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:33.494583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.494553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:33.766845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.766817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-583a616d"]
Sep 18 19:36:33.768983 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:36:33.768952    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a4b7af_d281_4090_9e4d_3dee18428990.slice/crio-a937e1ebf7152f528177bf8b79462c179c919abf4b5479cdae52e6cdfecd5932 WatchSource:0}: Error finding container a937e1ebf7152f528177bf8b79462c179c919abf4b5479cdae52e6cdfecd5932: Status 404 returned error can't find the container with id a937e1ebf7152f528177bf8b79462c179c919abf4b5479cdae52e6cdfecd5932
Sep 18 19:36:33.770954 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:33.770937    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:36:34.436559 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:34.436524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-583a616d" event={"ID":"60a4b7af-d281-4090-9e4d-3dee18428990","Type":"ContainerStarted","Data":"a937e1ebf7152f528177bf8b79462c179c919abf4b5479cdae52e6cdfecd5932"}
Sep 18 19:36:35.442083 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:35.442047    2583 generic.go:356] "Generic (PLEG): container finished" podID="60a4b7af-d281-4090-9e4d-3dee18428990" containerID="52dceae3b7c5a9b06f6581767a2a06bebffb001cdea31f61b4cf555de11eca6a" exitCode=0
Sep 18 19:36:35.442490 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:35.442132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-583a616d" event={"ID":"60a4b7af-d281-4090-9e4d-3dee18428990","Type":"ContainerDied","Data":"52dceae3b7c5a9b06f6581767a2a06bebffb001cdea31f61b4cf555de11eca6a"}
Sep 18 19:36:36.569678 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.569655    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:36.614467 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.614433    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-583a616d_60a4b7af-d281-4090-9e4d-3dee18428990/curl-metrics-583a616d/0.log"
Sep 18 19:36:36.638177 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.638141    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-583a616d"]
Sep 18 19:36:36.640215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.640187    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-583a616d"]
Sep 18 19:36:36.697684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.697586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwgqq\" (UniqueName: \"kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq\") pod \"60a4b7af-d281-4090-9e4d-3dee18428990\" (UID: \"60a4b7af-d281-4090-9e4d-3dee18428990\") "
Sep 18 19:36:36.699696 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.699657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq" (OuterVolumeSpecName: "kube-api-access-qwgqq") pod "60a4b7af-d281-4090-9e4d-3dee18428990" (UID: "60a4b7af-d281-4090-9e4d-3dee18428990"). InnerVolumeSpecName "kube-api-access-qwgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:36.798393 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:36.798323    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwgqq\" (UniqueName: \"kubernetes.io/projected/60a4b7af-d281-4090-9e4d-3dee18428990-kube-api-access-qwgqq\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:37.451459 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:37.451425    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:37.451459 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:37.451455    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a937e1ebf7152f528177bf8b79462c179c919abf4b5479cdae52e6cdfecd5932"
Sep 18 19:36:37.491259 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:36:37.491216    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-583a616d\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="60a4b7af-d281-4090-9e4d-3dee18428990" pod="kserve/curl-metrics-583a616d"
Sep 18 19:36:37.791660 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:37.791627    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a4b7af-d281-4090-9e4d-3dee18428990" path="/var/lib/kubelet/pods/60a4b7af-d281-4090-9e4d-3dee18428990/volumes"
Sep 18 19:36:41.999817 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:41.999727    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-67221731"]
Sep 18 19:36:42.000184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.000029    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a4b7af-d281-4090-9e4d-3dee18428990" containerName="curl-metrics-583a616d"
Sep 18 19:36:42.000184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.000040    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a4b7af-d281-4090-9e4d-3dee18428990" containerName="curl-metrics-583a616d"
Sep 18 19:36:42.000184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.000112    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a4b7af-d281-4090-9e4d-3dee18428990" containerName="curl-metrics-583a616d"
Sep 18 19:36:42.003229 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.003208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-67221731"
Sep 18 19:36:42.006441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.006418    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:36:42.007534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.007515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:36:42.007534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.007522    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8gwl5\""
Sep 18 19:36:42.012411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.012387    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-67221731"]
Sep 18 19:36:42.143930 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.143892    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc492\" (UniqueName: \"kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492\") pod \"curl-metrics-67221731\" (UID: \"0b6d4190-55b6-43b7-8e6e-e8980c04b337\") " pod="kserve/curl-metrics-67221731"
Sep 18 19:36:42.244758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.244714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc492\" (UniqueName: \"kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492\") pod \"curl-metrics-67221731\" (UID: \"0b6d4190-55b6-43b7-8e6e-e8980c04b337\") " pod="kserve/curl-metrics-67221731"
Sep 18 19:36:42.254451 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.254383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc492\" (UniqueName: \"kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492\") pod \"curl-metrics-67221731\" (UID: \"0b6d4190-55b6-43b7-8e6e-e8980c04b337\") " pod="kserve/curl-metrics-67221731"
Sep 18 19:36:42.327109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.327074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-67221731"
Sep 18 19:36:42.598035 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:42.598008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-67221731"]
Sep 18 19:36:42.600778 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:36:42.600741    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b6d4190_55b6_43b7_8e6e_e8980c04b337.slice/crio-35b0445316870f66c783858940f210ddaf648e289f2f6bd1480b33badc6524ac WatchSource:0}: Error finding container 35b0445316870f66c783858940f210ddaf648e289f2f6bd1480b33badc6524ac: Status 404 returned error can't find the container with id 35b0445316870f66c783858940f210ddaf648e289f2f6bd1480b33badc6524ac
Sep 18 19:36:43.479608 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:43.479570    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" containerID="699a568a5c263e04b9e5a215ddb74986d25648f5ebdae059b0f89a65b38372fe" exitCode=0
Sep 18 19:36:43.479608 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:43.479613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-67221731" event={"ID":"0b6d4190-55b6-43b7-8e6e-e8980c04b337","Type":"ContainerDied","Data":"699a568a5c263e04b9e5a215ddb74986d25648f5ebdae059b0f89a65b38372fe"}
Sep 18 19:36:43.480085 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:43.479641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-67221731" event={"ID":"0b6d4190-55b6-43b7-8e6e-e8980c04b337","Type":"ContainerStarted","Data":"35b0445316870f66c783858940f210ddaf648e289f2f6bd1480b33badc6524ac"}
Sep 18 19:36:44.614081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.614055    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-67221731"
Sep 18 19:36:44.635650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.635621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-67221731_0b6d4190-55b6-43b7-8e6e-e8980c04b337/curl-metrics-67221731/0.log"
Sep 18 19:36:44.657854 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.657820    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-67221731"]
Sep 18 19:36:44.660202 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.660179    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-67221731"]
Sep 18 19:36:44.665001 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.664979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc492\" (UniqueName: \"kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492\") pod \"0b6d4190-55b6-43b7-8e6e-e8980c04b337\" (UID: \"0b6d4190-55b6-43b7-8e6e-e8980c04b337\") "
Sep 18 19:36:44.667035 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.667008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492" (OuterVolumeSpecName: "kube-api-access-wc492") pod "0b6d4190-55b6-43b7-8e6e-e8980c04b337" (UID: "0b6d4190-55b6-43b7-8e6e-e8980c04b337"). InnerVolumeSpecName "kube-api-access-wc492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:36:44.766566 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:44.766531    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc492\" (UniqueName: \"kubernetes.io/projected/0b6d4190-55b6-43b7-8e6e-e8980c04b337-kube-api-access-wc492\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:36:45.489417 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:45.489388    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-67221731"
Sep 18 19:36:45.489587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:45.489393    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b0445316870f66c783858940f210ddaf648e289f2f6bd1480b33badc6524ac"
Sep 18 19:36:45.499553 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:36:45.499521    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-67221731\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" pod="kserve/curl-metrics-67221731"
Sep 18 19:36:45.791454 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:36:45.791416    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" path="/var/lib/kubelet/pods/0b6d4190-55b6-43b7-8e6e-e8980c04b337/volumes"
Sep 18 19:38:56.106842 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.106802    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:38:56.107491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.107134    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" containerName="curl-metrics-67221731"
Sep 18 19:38:56.107491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.107146    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" containerName="curl-metrics-67221731"
Sep 18 19:38:56.107491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.107209    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6d4190-55b6-43b7-8e6e-e8980c04b337" containerName="curl-metrics-67221731"
Sep 18 19:38:56.110283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.110264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.113713 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113689    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:38:56.113834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:38:56.113834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.113834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.113834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 18 19:38:56.113834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:38:56.114116 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.113690    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 19:38:56.124320 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.124296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:38:56.275433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.275393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.275433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.275436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.275698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.275474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.275698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.275555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrjz\" (UniqueName: \"kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.275698 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.275585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.376815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.376699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrjz\" (UniqueName: \"kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.376815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.376763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.376825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377053 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:38:56.376937    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:38:56.377053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.376947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377053 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:38:56.377001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls podName:15ab049c-c3e7-42c8-8f9d-e9e2ed00a496 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:56.876979656 +0000 UTC m=+1469.252405941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls") pod "sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:38:56.377274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.377078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377274 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.377256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377576 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.377556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.377661 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.377644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.387678 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.387657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrjz\" (UniqueName: \"kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.881467 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:56.881425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:56.881643 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:38:56.881576    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:38:56.881689 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:38:56.881653    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls podName:15ab049c-c3e7-42c8-8f9d-e9e2ed00a496 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.881627367 +0000 UTC m=+1470.257053643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls") pod "sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:38:57.890667 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:57.890623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:57.893157 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:57.893127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:57.920085 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:57.920046    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:38:58.203157 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:58.203128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:38:58.205967 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:38:58.205938    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ab049c_c3e7_42c8_8f9d_e9e2ed00a496.slice/crio-0bebf1d52ed3a220553d3197c320d57920287b383d2e46c695cc4d08deec7bee WatchSource:0}: Error finding container 0bebf1d52ed3a220553d3197c320d57920287b383d2e46c695cc4d08deec7bee: Status 404 returned error can't find the container with id 0bebf1d52ed3a220553d3197c320d57920287b383d2e46c695cc4d08deec7bee
Sep 18 19:38:59.050803 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:59.050771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerStarted","Data":"449282a1ea785153bc472d604ecbb4beee35a6e5bf510bb6281572ae4b82c687"}
Sep 18 19:38:59.051208 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:38:59.050823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerStarted","Data":"0bebf1d52ed3a220553d3197c320d57920287b383d2e46c695cc4d08deec7bee"}
Sep 18 19:39:02.065201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:39:02.065161    2583 generic.go:356] "Generic (PLEG): container finished" podID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerID="449282a1ea785153bc472d604ecbb4beee35a6e5bf510bb6281572ae4b82c687" exitCode=0
Sep 18 19:39:02.065627 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:39:02.065231    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerDied","Data":"449282a1ea785153bc472d604ecbb4beee35a6e5bf510bb6281572ae4b82c687"}
Sep 18 19:39:29.801547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:39:29.801513    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:39:29.804644 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:39:29.804621    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:41:03.691465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:03.691412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerStarted","Data":"57a8e669d114a5b71feeca9ab7980bae8ef9fa71b957aedc4cf23364f506b93c"}
Sep 18 19:41:03.691925 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:03.691489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerStarted","Data":"eedbba525d2f825cedf7bb41a276fc635c81e6c110b1470eaa4934a1c672d365"}
Sep 18 19:41:03.691925 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:03.691525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:41:03.717814 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:03.717759    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podStartSLOduration=3.274595714 podStartE2EDuration="2m7.717741156s" totalImagesPullingTime="2m0.700776514s" totalInitContainerRuntime="3.742368928s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:03.716234111 +0000 UTC m=+1596.091660405" watchObservedRunningTime="2026-09-18 19:41:03.717741156 +0000 UTC m=+1596.093167449"
Sep 18 19:41:04.696803 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:04.696771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:41:10.708622 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:10.708542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:41:40.799357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:40.799290    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 19:41:50.711732 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:50.711695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:41:53.804599 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.804567    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:41:53.807104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.807088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.810845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.810827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 19:41:53.811140 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.811116    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 19:41:53.845700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.845669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:41:53.856250 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.856224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.856353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.856260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.856353 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.856285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.856449 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.856383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k5v\" (UniqueName: \"kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.856449 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.856400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957054 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k5v\" (UniqueName: \"kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957728 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.957911 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.957893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.959711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.959689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:53.967257 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:53.967227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k5v\" (UniqueName: \"kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v\") pod \"isvc-canary-create-predictor-7b568f75-7j6hq\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:54.117534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:54.117435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:41:54.393028 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:54.393001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:41:54.394815 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:41:54.394782    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755d91c4_c322_4fcf_b312_212cd448d154.slice/crio-0fd7bed0b2de79ffc02a947304ec7428af6a6c1ea059d662dbb5025c8bffd615 WatchSource:0}: Error finding container 0fd7bed0b2de79ffc02a947304ec7428af6a6c1ea059d662dbb5025c8bffd615: Status 404 returned error can't find the container with id 0fd7bed0b2de79ffc02a947304ec7428af6a6c1ea059d662dbb5025c8bffd615
Sep 18 19:41:54.918931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:54.918893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerStarted","Data":"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"}
Sep 18 19:41:54.919318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:54.918954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerStarted","Data":"0fd7bed0b2de79ffc02a947304ec7428af6a6c1ea059d662dbb5025c8bffd615"}
Sep 18 19:41:56.288551 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:56.288514    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:41:56.288985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:56.288759    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container" containerID="cri-o://eedbba525d2f825cedf7bb41a276fc635c81e6c110b1470eaa4934a1c672d365" gracePeriod=30
Sep 18 19:41:56.288985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:56.288808    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy" containerID="cri-o://57a8e669d114a5b71feeca9ab7980bae8ef9fa71b957aedc4cf23364f506b93c" gracePeriod=30
Sep 18 19:41:56.929758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:56.929723    2583 generic.go:356] "Generic (PLEG): container finished" podID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerID="57a8e669d114a5b71feeca9ab7980bae8ef9fa71b957aedc4cf23364f506b93c" exitCode=2
Sep 18 19:41:56.929943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:56.929799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerDied","Data":"57a8e669d114a5b71feeca9ab7980bae8ef9fa71b957aedc4cf23364f506b93c"}
Sep 18 19:41:58.942819 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:58.942787    2583 generic.go:356] "Generic (PLEG): container finished" podID="755d91c4-c322-4fcf-b312-212cd448d154" containerID="b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a" exitCode=0
Sep 18 19:41:58.943279 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:58.942839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerDied","Data":"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"}
Sep 18 19:41:58.944215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:41:58.944194    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:42:00.702102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:00.702043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 19:42:00.709851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:00.709754    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 19:42:05.702106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:05.702058    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 19:42:08.000714 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:08.000658    2583 generic.go:356] "Generic (PLEG): container finished" podID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerID="eedbba525d2f825cedf7bb41a276fc635c81e6c110b1470eaa4934a1c672d365" exitCode=0
Sep 18 19:42:08.001247 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:08.000774    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerDied","Data":"eedbba525d2f825cedf7bb41a276fc635c81e6c110b1470eaa4934a1c672d365"}
Sep 18 19:42:11.703030 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:11.702982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:42:11.703570 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:11.703130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:42:11.710107 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:11.710068    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:42:12.263506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.263479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:42:12.429230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429201    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") pod \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") "
Sep 18 19:42:12.429386 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429258    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location\") pod \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") "
Sep 18 19:42:12.429386 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config\") pod \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") "
Sep 18 19:42:12.429386 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert\") pod \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") "
Sep 18 19:42:12.429386 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqrjz\" (UniqueName: \"kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz\") pod \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\" (UID: \"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496\") "
Sep 18 19:42:12.429693 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:12.429693 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:42:12.429836 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.429666    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:42:12.431288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.431266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:42:12.431348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.431312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz" (OuterVolumeSpecName: "kube-api-access-hqrjz") pod "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" (UID: "15ab049c-c3e7-42c8-8f9d-e9e2ed00a496"). InnerVolumeSpecName "kube-api-access-hqrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:42:12.530604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.530514    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:42:12.530604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.530551    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:42:12.530604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.530562    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqrjz\" (UniqueName: \"kubernetes.io/projected/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kube-api-access-hqrjz\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:42:12.530604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.530573    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:42:12.530604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:12.530582    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:42:13.028233 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.028199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m" event={"ID":"15ab049c-c3e7-42c8-8f9d-e9e2ed00a496","Type":"ContainerDied","Data":"0bebf1d52ed3a220553d3197c320d57920287b383d2e46c695cc4d08deec7bee"}
Sep 18 19:42:13.028763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.028267    2583 scope.go:117] "RemoveContainer" containerID="57a8e669d114a5b71feeca9ab7980bae8ef9fa71b957aedc4cf23364f506b93c"
Sep 18 19:42:13.028763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.028208    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"
Sep 18 19:42:13.030620 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.030594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerStarted","Data":"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"}
Sep 18 19:42:13.030762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.030647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerStarted","Data":"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"}
Sep 18 19:42:13.030855 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.030837    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:42:13.030916 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.030893    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:42:13.032530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.032501    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:13.037251 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.037236    2583 scope.go:117] "RemoveContainer" containerID="eedbba525d2f825cedf7bb41a276fc635c81e6c110b1470eaa4934a1c672d365"
Sep 18 19:42:13.045583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.045566    2583 scope.go:117] "RemoveContainer" containerID="449282a1ea785153bc472d604ecbb4beee35a6e5bf510bb6281572ae4b82c687"
Sep 18 19:42:13.054039 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.054011    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:42:13.058524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.058503    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-55fb4d8685-j7p9m"]
Sep 18 19:42:13.086613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.086562    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podStartSLOduration=2.697183232 podStartE2EDuration="20.086546055s" totalImagesPullingTime="13.392682298s" totalInitContainerRuntime="3.996680525s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:13.083804884 +0000 UTC m=+1665.459231175" watchObservedRunningTime="2026-09-18 19:42:13.086546055 +0000 UTC m=+1665.461972348"
Sep 18 19:42:13.791727 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:13.791695    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" path="/var/lib/kubelet/pods/15ab049c-c3e7-42c8-8f9d-e9e2ed00a496/volumes"
Sep 18 19:42:14.039124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:14.039089    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:19.044058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:19.044029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:42:19.044575 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:19.044548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:29.045372 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:29.045314    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:38.985993 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.985952    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986284    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986298    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986316    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986322    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986339    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="storage-initializer"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986345    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="storage-initializer"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986407    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kube-rbac-proxy"
Sep 18 19:42:38.986422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:38.986418    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ab049c-c3e7-42c8-8f9d-e9e2ed00a496" containerName="kserve-container"
Sep 18 19:42:39.044675 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.044634    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:39.352846 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.352801    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:42:39.353029 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.352918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.363237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.363212    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:42:39.363354 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.363257    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 18 19:42:39.459721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.459675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.459721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.459720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.459950 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.459811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.459950 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.459849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.459950 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.459901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgcjw\" (UniqueName: \"kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.560703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.560666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.560703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.560704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.560960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.560745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgcjw\" (UniqueName: \"kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.560960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.560790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.560960 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.560807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.561156 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.561136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.561447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.561430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.561546 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.561528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.563185 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.563164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.571690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.571663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgcjw\" (UniqueName: \"kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw\") pod \"isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.663070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.662962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:39.998585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:39.998554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:42:40.001144 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:42:40.001109    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b1188e_f5d6_4e42_a0bf_5b9be6c10805.slice/crio-e5109fb860d659c8e355cc8a49b383075d625aaf31e89224698036fbe26dca24 WatchSource:0}: Error finding container e5109fb860d659c8e355cc8a49b383075d625aaf31e89224698036fbe26dca24: Status 404 returned error can't find the container with id e5109fb860d659c8e355cc8a49b383075d625aaf31e89224698036fbe26dca24
Sep 18 19:42:40.157968 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:40.157931    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerStarted","Data":"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"}
Sep 18 19:42:40.158313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:40.158004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerStarted","Data":"e5109fb860d659c8e355cc8a49b383075d625aaf31e89224698036fbe26dca24"}
Sep 18 19:42:44.175547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:44.175509    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerID="f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0" exitCode=0
Sep 18 19:42:44.175948 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:44.175587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerDied","Data":"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"}
Sep 18 19:42:45.182486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:45.182453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerStarted","Data":"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"}
Sep 18 19:42:45.182896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:45.182516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerStarted","Data":"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"}
Sep 18 19:42:45.182896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:45.182808    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:45.182896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:45.182871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:45.218409 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:45.218332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podStartSLOduration=3.262239323 podStartE2EDuration="7.218314883s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.95607556s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:45.216465816 +0000 UTC m=+1697.591892109" watchObservedRunningTime="2026-09-18 19:42:45.218314883 +0000 UTC m=+1697.593741176"
Sep 18 19:42:49.045521 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:49.045472    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:42:51.193965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:51.193937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:42:59.044806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:42:59.044761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:43:09.045416 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:09.045354    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:43:19.044817 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:19.044769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:43:21.199980 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:21.199927    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 19:43:26.814719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:26.814683    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:43:26.815181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:26.814912    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" containerID="cri-o://02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c" gracePeriod=30
Sep 18 19:43:26.815181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:26.814974    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" containerID="cri-o://7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9" gracePeriod=30
Sep 18 19:43:26.815313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:26.815188    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 19:43:27.372146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:27.372112    2583 generic.go:356] "Generic (PLEG): container finished" podID="755d91c4-c322-4fcf-b312-212cd448d154" containerID="7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9" exitCode=2
Sep 18 19:43:27.372323 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:27.372188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerDied","Data":"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"}
Sep 18 19:43:29.039700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:29.039652    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:29.875735 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:29.875701    2583 scope.go:117] "RemoveContainer" containerID="699a568a5c263e04b9e5a215ddb74986d25648f5ebdae059b0f89a65b38372fe"
Sep 18 19:43:29.883473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:29.883449    2583 scope.go:117] "RemoveContainer" containerID="52dceae3b7c5a9b06f6581767a2a06bebffb001cdea31f61b4cf555de11eca6a"
Sep 18 19:43:31.196873 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:31.196843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:43:34.039506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:34.039462    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:36.817169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:36.817137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:43:38.776288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.776245    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:43:38.781282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.781259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.787774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.787728    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 19:43:38.788317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.788298    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 19:43:38.803524 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.803502    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:43:38.860684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.860654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.860684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.860688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.860871 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.860787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.860871 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.860809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw6g\" (UniqueName: \"kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.860951 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.860882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.920471 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.920442    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:43:38.924079 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.924055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.928534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.928511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 19:43:38.962193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzjb\" (UniqueName: \"kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.962193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962516 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw6g\" (UniqueName: \"kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.962780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:38.962780 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:38.962615    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 18 19:43:38.962780 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:38.962694    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls podName:40858d80-4a8c-4061-a42c-79db3b2c9e67 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:39.462674312 +0000 UTC m=+1751.838100599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls") pod "isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 18 19:43:38.962780 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.962966 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.962954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:38.973522 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.973494    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:43:38.985787 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:38.985761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw6g\" (UniqueName: \"kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:39.039827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.039725    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:39.039991 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.039870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:43:39.063596 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.063557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.063799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.063608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.063799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.063644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.063799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.063663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.063799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.063698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzjb\" (UniqueName: \"kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.064181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.064146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.064343 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.064323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.064465 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.064445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.066094 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.066069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.076114 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.076082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzjb\" (UniqueName: \"kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb\") pod \"isvc-canary-promote-predictor-75f64ff6c9-sqmh9\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.158124 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.158087    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:43:39.159024 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.158353    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container" containerID="cri-o://a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79" gracePeriod=30
Sep 18 19:43:39.159024 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.158424    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy" containerID="cri-o://743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54" gracePeriod=30
Sep 18 19:43:39.234705 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.234661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:39.425194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.425102    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerID="743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54" exitCode=2
Sep 18 19:43:39.425194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.425154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerDied","Data":"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"}
Sep 18 19:43:39.466783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.466746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:39.469162 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.469136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:39.522228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.522198    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:43:39.524838 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:43:39.524806    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f4caf0_6585_4b3b_ab7f_3b3c3cc226a4.slice/crio-e602caabedd332b5b466ba8bc37cffe3df66ef664892366c3a79ce2b873ba657 WatchSource:0}: Error finding container e602caabedd332b5b466ba8bc37cffe3df66ef664892366c3a79ce2b873ba657: Status 404 returned error can't find the container with id e602caabedd332b5b466ba8bc37cffe3df66ef664892366c3a79ce2b873ba657
Sep 18 19:43:39.691649 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.691543    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:39.973001 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:39.972797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:43:39.975437 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:43:39.975409    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40858d80_4a8c_4061_a42c_79db3b2c9e67.slice/crio-62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e WatchSource:0}: Error finding container 62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e: Status 404 returned error can't find the container with id 62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e
Sep 18 19:43:40.431641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:40.431598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerStarted","Data":"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"}
Sep 18 19:43:40.431843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:40.431700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerStarted","Data":"62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e"}
Sep 18 19:43:40.433228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:40.433200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerStarted","Data":"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"}
Sep 18 19:43:40.433355 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:40.433256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerStarted","Data":"e602caabedd332b5b466ba8bc37cffe3df66ef664892366c3a79ce2b873ba657"}
Sep 18 19:43:41.188292 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:41.188242    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 19:43:42.236566 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:42.236507    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 19:43:44.040243 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:44.040197    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:44.452136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:44.452104    2583 generic.go:356] "Generic (PLEG): container finished" podID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerID="cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673" exitCode=0
Sep 18 19:43:44.452331 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:44.452185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerDied","Data":"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"}
Sep 18 19:43:44.453902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:44.453882    2583 generic.go:356] "Generic (PLEG): container finished" podID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerID="34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496" exitCode=0
Sep 18 19:43:44.454015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:44.453911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerDied","Data":"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"}
Sep 18 19:43:45.461469 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.461433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerStarted","Data":"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"}
Sep 18 19:43:45.461932 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.461511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerStarted","Data":"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"}
Sep 18 19:43:45.461932 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.461755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:45.463834 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.463813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerStarted","Data":"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"}
Sep 18 19:43:45.463943 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.463859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerStarted","Data":"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"}
Sep 18 19:43:45.464053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.464026    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:45.464185 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.464071    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:45.465717 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.465692    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:43:45.487715 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.487654    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podStartSLOduration=3.52225559 podStartE2EDuration="7.487637117s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.965381527s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:45.485578406 +0000 UTC m=+1757.861004698" watchObservedRunningTime="2026-09-18 19:43:45.487637117 +0000 UTC m=+1757.863063467"
Sep 18 19:43:45.518525 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:45.518453    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podStartSLOduration=3.583764781 podStartE2EDuration="7.518436346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934671565s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:45.515106805 +0000 UTC m=+1757.890533098" watchObservedRunningTime="2026-09-18 19:43:45.518436346 +0000 UTC m=+1757.893862638"
Sep 18 19:43:46.187976 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:46.187924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 19:43:46.469037 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:46.468933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:46.469037 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:46.468985    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:43:46.470261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:46.470235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:43:47.008850 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.008825    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:43:47.139945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.139860    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls\") pod \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") "
Sep 18 19:43:47.139945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.139910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert\") pod \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") "
Sep 18 19:43:47.139945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.139939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config\") pod \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") "
Sep 18 19:43:47.140170 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.140056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location\") pod \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") "
Sep 18 19:43:47.140170 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.140113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgcjw\" (UniqueName: \"kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw\") pod \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\" (UID: \"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805\") "
Sep 18 19:43:47.140378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.140336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" (UID: "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:47.140470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.140389    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" (UID: "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:47.140470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.140391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" (UID: "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:47.142083 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.142054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" (UID: "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:47.142184 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.142143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw" (OuterVolumeSpecName: "kube-api-access-mgcjw") pod "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" (UID: "c4b1188e-f5d6-4e42-a0bf-5b9be6c10805"). InnerVolumeSpecName "kube-api-access-mgcjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:47.241136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.241101    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:47.241136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.241128    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:47.241136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.241137    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:47.241394 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.241147    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:47.241394 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.241156    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgcjw\" (UniqueName: \"kubernetes.io/projected/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805-kube-api-access-mgcjw\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:47.476744 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.476652    2583 generic.go:356] "Generic (PLEG): container finished" podID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerID="a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79" exitCode=0
Sep 18 19:43:47.476744 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.476714    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"
Sep 18 19:43:47.476744 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.476724    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerDied","Data":"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"}
Sep 18 19:43:47.477298 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.476809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8" event={"ID":"c4b1188e-f5d6-4e42-a0bf-5b9be6c10805","Type":"ContainerDied","Data":"e5109fb860d659c8e355cc8a49b383075d625aaf31e89224698036fbe26dca24"}
Sep 18 19:43:47.477298 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.476846    2583 scope.go:117] "RemoveContainer" containerID="743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"
Sep 18 19:43:47.477431 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.477379    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:43:47.486481 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.486462    2583 scope.go:117] "RemoveContainer" containerID="a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"
Sep 18 19:43:47.494805 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.494781    2583 scope.go:117] "RemoveContainer" containerID="f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"
Sep 18 19:43:47.504339 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.504304    2583 scope.go:117] "RemoveContainer" containerID="743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"
Sep 18 19:43:47.504819 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:47.504790    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54\": container with ID starting with 743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54 not found: ID does not exist" containerID="743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"
Sep 18 19:43:47.504931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.504832    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54"} err="failed to get container status \"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54\": rpc error: code = NotFound desc = could not find container \"743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54\": container with ID starting with 743230d257478bfa7f5a9a74ac43993274e844b22b2d0f49a80b09f37575ff54 not found: ID does not exist"
Sep 18 19:43:47.504931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.504857    2583 scope.go:117] "RemoveContainer" containerID="a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"
Sep 18 19:43:47.505285 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:47.505258    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79\": container with ID starting with a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79 not found: ID does not exist" containerID="a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"
Sep 18 19:43:47.505439 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.505291    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79"} err="failed to get container status \"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79\": rpc error: code = NotFound desc = could not find container \"a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79\": container with ID starting with a034a934cf16fa79b19f78db774cecb76dbb1de4f141a4b6c26b610593151b79 not found: ID does not exist"
Sep 18 19:43:47.505439 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.505314    2583 scope.go:117] "RemoveContainer" containerID="f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"
Sep 18 19:43:47.505669 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:47.505646    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0\": container with ID starting with f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0 not found: ID does not exist" containerID="f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"
Sep 18 19:43:47.505737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.505679    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0"} err="failed to get container status \"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0\": rpc error: code = NotFound desc = could not find container \"f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0\": container with ID starting with f0e9b905274b622328374f77ce4e4acd78644fd80e22781d4404c60a19352ce0 not found: ID does not exist"
Sep 18 19:43:47.506064 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.506045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:43:47.512147 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.512113    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-698d8768cc-6twc8"]
Sep 18 19:43:47.791450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:47.791418    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" path="/var/lib/kubelet/pods/c4b1188e-f5d6-4e42-a0bf-5b9be6c10805/volumes"
Sep 18 19:43:49.039902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:49.039856    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:51.473807 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:51.473776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:43:51.474279 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:51.474253    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:43:52.481946 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:52.481914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:43:52.482519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:52.482486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:43:54.039986 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:54.039936    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 19:43:57.460390 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.460351    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:43:57.527559 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527520    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location\") pod \"755d91c4-c322-4fcf-b312-212cd448d154\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") "
Sep 18 19:43:57.527559 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527561    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config\") pod \"755d91c4-c322-4fcf-b312-212cd448d154\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") "
Sep 18 19:43:57.527831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527583    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5k5v\" (UniqueName: \"kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v\") pod \"755d91c4-c322-4fcf-b312-212cd448d154\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") "
Sep 18 19:43:57.527831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert\") pod \"755d91c4-c322-4fcf-b312-212cd448d154\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") "
Sep 18 19:43:57.527831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527700    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls\") pod \"755d91c4-c322-4fcf-b312-212cd448d154\" (UID: \"755d91c4-c322-4fcf-b312-212cd448d154\") "
Sep 18 19:43:57.527999 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.527912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "755d91c4-c322-4fcf-b312-212cd448d154" (UID: "755d91c4-c322-4fcf-b312-212cd448d154"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:57.528084 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528053    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "755d91c4-c322-4fcf-b312-212cd448d154" (UID: "755d91c4-c322-4fcf-b312-212cd448d154"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:57.528143 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "755d91c4-c322-4fcf-b312-212cd448d154" (UID: "755d91c4-c322-4fcf-b312-212cd448d154"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:57.528714 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528680    2583 generic.go:356] "Generic (PLEG): container finished" podID="755d91c4-c322-4fcf-b312-212cd448d154" containerID="02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c" exitCode=137
Sep 18 19:43:57.528806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528791    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"
Sep 18 19:43:57.528852 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerDied","Data":"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"}
Sep 18 19:43:57.528914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq" event={"ID":"755d91c4-c322-4fcf-b312-212cd448d154","Type":"ContainerDied","Data":"0fd7bed0b2de79ffc02a947304ec7428af6a6c1ea059d662dbb5025c8bffd615"}
Sep 18 19:43:57.528965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.528951    2583 scope.go:117] "RemoveContainer" containerID="7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"
Sep 18 19:43:57.530121 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.530101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v" (OuterVolumeSpecName: "kube-api-access-m5k5v") pod "755d91c4-c322-4fcf-b312-212cd448d154" (UID: "755d91c4-c322-4fcf-b312-212cd448d154"). InnerVolumeSpecName "kube-api-access-m5k5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:57.530201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.530144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "755d91c4-c322-4fcf-b312-212cd448d154" (UID: "755d91c4-c322-4fcf-b312-212cd448d154"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:57.545594 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.545573    2583 scope.go:117] "RemoveContainer" containerID="02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"
Sep 18 19:43:57.553677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.553656    2583 scope.go:117] "RemoveContainer" containerID="b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"
Sep 18 19:43:57.561461 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.561436    2583 scope.go:117] "RemoveContainer" containerID="7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"
Sep 18 19:43:57.561804 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:57.561782    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9\": container with ID starting with 7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9 not found: ID does not exist" containerID="7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"
Sep 18 19:43:57.561860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.561814    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9"} err="failed to get container status \"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9\": rpc error: code = NotFound desc = could not find container \"7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9\": container with ID starting with 7bd4daa0339fecc55b830a9d94aa3b2b45e2180616b2c6263e9519cc9e449aa9 not found: ID does not exist"
Sep 18 19:43:57.561860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.561833    2583 scope.go:117] "RemoveContainer" containerID="02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"
Sep 18 19:43:57.562121 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:57.562104    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c\": container with ID starting with 02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c not found: ID does not exist" containerID="02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"
Sep 18 19:43:57.562174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.562125    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c"} err="failed to get container status \"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c\": rpc error: code = NotFound desc = could not find container \"02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c\": container with ID starting with 02c4acca2e79ce1cefa8516ac8e3d77fcb6ec0eda33dacc13bd8714f5168389c not found: ID does not exist"
Sep 18 19:43:57.562174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.562138    2583 scope.go:117] "RemoveContainer" containerID="b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"
Sep 18 19:43:57.562408 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:43:57.562386    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a\": container with ID starting with b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a not found: ID does not exist" containerID="b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"
Sep 18 19:43:57.562461 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.562413    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a"} err="failed to get container status \"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a\": rpc error: code = NotFound desc = could not find container \"b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a\": container with ID starting with b18249a7037f5baf5b803f6535420889b1ade992545e93f2acc03d5a42b68a0a not found: ID does not exist"
Sep 18 19:43:57.628944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.628849    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/755d91c4-c322-4fcf-b312-212cd448d154-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:57.628944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.628881    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:57.628944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.628894    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k5v\" (UniqueName: \"kubernetes.io/projected/755d91c4-c322-4fcf-b312-212cd448d154-kube-api-access-m5k5v\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:57.628944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.628903    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/755d91c4-c322-4fcf-b312-212cd448d154-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:57.628944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.628912    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/755d91c4-c322-4fcf-b312-212cd448d154-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:43:57.847397 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.847351    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:43:57.850026 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:57.849998    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7b568f75-7j6hq"]
Sep 18 19:43:59.792119 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:43:59.792088    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="755d91c4-c322-4fcf-b312-212cd448d154" path="/var/lib/kubelet/pods/755d91c4-c322-4fcf-b312-212cd448d154/volumes"
Sep 18 19:44:01.474959 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:01.474913    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:02.482496 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:02.482454    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:44:11.475238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:11.475151    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:12.482587 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:12.482545    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:44:21.474551 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:21.474510    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:22.482882 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:22.482844    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:44:29.835425 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:29.835397    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:44:29.838944 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:29.838919    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:44:31.474418 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:31.474376    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:32.482736 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:32.482697    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:44:41.474323 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:41.474276    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:42.482574 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:42.482527    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:44:51.474644 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:51.474604    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:44:52.482434 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:44:52.482391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:45:01.474958 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:01.474917    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 19:45:02.482718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:02.482665    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:45:03.787263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:03.787217    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:45:07.792068 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:07.792038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:45:13.791102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:13.791074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:45:18.747502 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:18.747464    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:45:18.747965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:18.747691    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" containerID="cri-o://7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0" gracePeriod=30
Sep 18 19:45:18.747965 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:18.747758    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kube-rbac-proxy" containerID="cri-o://4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a" gracePeriod=30
Sep 18 19:45:18.893373 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:18.893335    2583 generic.go:356] "Generic (PLEG): container finished" podID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerID="4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a" exitCode=2
Sep 18 19:45:18.893563 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:18.893412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerDied","Data":"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"}
Sep 18 19:45:20.066082 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066035    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:45:20.066595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066542    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container"
Sep 18 19:45:20.066595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066563    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container"
Sep 18 19:45:20.066595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066575    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066584    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066595 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066595    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066603    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066629    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="storage-initializer"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066639    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="storage-initializer"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066668    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="storage-initializer"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066678    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="storage-initializer"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066688    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066698    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066781    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066794    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kube-rbac-proxy"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066804    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="755d91c4-c322-4fcf-b312-212cd448d154" containerName="kserve-container"
Sep 18 19:45:20.066883 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.066818    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b1188e-f5d6-4e42-a0bf-5b9be6c10805" containerName="kserve-container"
Sep 18 19:45:20.070238 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.070215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.073088 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.073063    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 18 19:45:20.073352 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.073339    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 18 19:45:20.085637 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.085609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:45:20.154486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.154448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfj28\" (UniqueName: \"kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.154690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.154502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.154690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.154554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.154690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.154604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.154690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.154668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255237 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfj28\" (UniqueName: \"kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255661 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:45:20.255476    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 18 19:45:20.255661 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:45:20.255549    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls podName:157aa436-fd40-46cd-870f-b1c02bba5f39 nodeName:}" failed. No retries permitted until 2026-09-18 19:45:20.755527326 +0000 UTC m=+1853.130953597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 18 19:45:20.255740 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.255957 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.256000 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.255969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.275634 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.275602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfj28\" (UniqueName: \"kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.758881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.758837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.761413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.761382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:20.980416 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:20.980341    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:21.258985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:21.258958    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:45:21.260847 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:45:21.260814    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157aa436_fd40_46cd_870f_b1c02bba5f39.slice/crio-2ac29e27b30a4501376b12a577814f43f5a7596a03af1e01a793ed02a5f79fa4 WatchSource:0}: Error finding container 2ac29e27b30a4501376b12a577814f43f5a7596a03af1e01a793ed02a5f79fa4: Status 404 returned error can't find the container with id 2ac29e27b30a4501376b12a577814f43f5a7596a03af1e01a793ed02a5f79fa4
Sep 18 19:45:21.908895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:21.908857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerStarted","Data":"3436df55843915a916976cd59b51b362289bca18a83f0c98e5255ce59db521e6"}
Sep 18 19:45:21.909153 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:21.908922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerStarted","Data":"2ac29e27b30a4501376b12a577814f43f5a7596a03af1e01a793ed02a5f79fa4"}
Sep 18 19:45:22.477827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:22.477780    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 19:45:23.788037 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:23.787994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 19:45:24.088429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.088396    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:45:24.194985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.194946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert\") pod \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") "
Sep 18 19:45:24.195161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195004    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wzjb\" (UniqueName: \"kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb\") pod \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") "
Sep 18 19:45:24.195161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls\") pod \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") "
Sep 18 19:45:24.195161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195087    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location\") pod \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") "
Sep 18 19:45:24.195161 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config\") pod \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\" (UID: \"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4\") "
Sep 18 19:45:24.195459 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195428    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" (UID: "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:24.195523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" (UID: "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:24.195584 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.195561    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" (UID: "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:45:24.197206 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.197185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb" (OuterVolumeSpecName: "kube-api-access-5wzjb") pod "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" (UID: "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4"). InnerVolumeSpecName "kube-api-access-5wzjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:24.197206 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.197191    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" (UID: "d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:45:24.295719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.295683    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:45:24.295719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.295711    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzjb\" (UniqueName: \"kubernetes.io/projected/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-api-access-5wzjb\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:45:24.295719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.295722    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:45:24.295959 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.295733    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:45:24.295959 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.295744    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:45:24.923778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.923731    2583 generic.go:356] "Generic (PLEG): container finished" podID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerID="7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0" exitCode=0
Sep 18 19:45:24.924204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.923792    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"
Sep 18 19:45:24.924204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.923807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerDied","Data":"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"}
Sep 18 19:45:24.924204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.923878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9" event={"ID":"d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4","Type":"ContainerDied","Data":"e602caabedd332b5b466ba8bc37cffe3df66ef664892366c3a79ce2b873ba657"}
Sep 18 19:45:24.924204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.923906    2583 scope.go:117] "RemoveContainer" containerID="4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"
Sep 18 19:45:24.933207 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.933189    2583 scope.go:117] "RemoveContainer" containerID="7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"
Sep 18 19:45:24.941309 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.941286    2583 scope.go:117] "RemoveContainer" containerID="34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"
Sep 18 19:45:24.949406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.949384    2583 scope.go:117] "RemoveContainer" containerID="4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"
Sep 18 19:45:24.949763 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:45:24.949741    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a\": container with ID starting with 4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a not found: ID does not exist" containerID="4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"
Sep 18 19:45:24.949860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.949779    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a"} err="failed to get container status \"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a\": rpc error: code = NotFound desc = could not find container \"4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a\": container with ID starting with 4649c2c3c5d86130da694487d4f5151a1f04ceb8df8cba1b72bac6660579727a not found: ID does not exist"
Sep 18 19:45:24.949860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.949806    2583 scope.go:117] "RemoveContainer" containerID="7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"
Sep 18 19:45:24.950109 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:45:24.950090    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0\": container with ID starting with 7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0 not found: ID does not exist" containerID="7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"
Sep 18 19:45:24.950180 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.950118    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0"} err="failed to get container status \"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0\": rpc error: code = NotFound desc = could not find container \"7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0\": container with ID starting with 7db81c306a584db011507285ec03e291ffc0eb794cce6d86ef7e5fd5b2178dd0 not found: ID does not exist"
Sep 18 19:45:24.950180 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.950140    2583 scope.go:117] "RemoveContainer" containerID="34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"
Sep 18 19:45:24.950468 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:45:24.950446    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496\": container with ID starting with 34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496 not found: ID does not exist" containerID="34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"
Sep 18 19:45:24.950546 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.950474    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496"} err="failed to get container status \"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496\": rpc error: code = NotFound desc = could not find container \"34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496\": container with ID starting with 34f77fdc6a411a098149e2344c1e3ca31d1b0bb2a3a5d32e1298331def85b496 not found: ID does not exist"
Sep 18 19:45:24.950680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.950660    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:45:24.961972 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:24.961938    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-75f64ff6c9-sqmh9"]
Sep 18 19:45:25.792011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:25.791977    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" path="/var/lib/kubelet/pods/d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4/volumes"
Sep 18 19:45:25.930070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:25.930038    2583 generic.go:356] "Generic (PLEG): container finished" podID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerID="3436df55843915a916976cd59b51b362289bca18a83f0c98e5255ce59db521e6" exitCode=0
Sep 18 19:45:25.930502 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:25.930111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerDied","Data":"3436df55843915a916976cd59b51b362289bca18a83f0c98e5255ce59db521e6"}
Sep 18 19:45:26.939731 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:26.939697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerStarted","Data":"e7e19a8e0b226f2a2e7006d0504bb11175e9bbbec33dfb817ff12db22f938d11"}
Sep 18 19:45:26.940199 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:26.939769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerStarted","Data":"200288bd275d3ec143fc1ca0a078505cd68c4c5a6feabc1377f252875e4fbf44"}
Sep 18 19:45:26.940199 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:26.939995    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:26.967674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:26.967616    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podStartSLOduration=3.16854152 podStartE2EDuration="6.967598232s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799056712s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:26.965437224 +0000 UTC m=+1859.340863517" watchObservedRunningTime="2026-09-18 19:45:26.967598232 +0000 UTC m=+1859.343024525"
Sep 18 19:45:27.948907 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:27.947442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:27.949392 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:27.949053    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:45:28.950557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:28.950517    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:45:33.955426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:33.955394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:45:33.955907 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:33.955879    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:45:43.956147 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:43.956058    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:45:53.956527 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:45:53.956474    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:03.956792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:03.956747    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:13.956775 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:13.956733    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:23.956823 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:23.956781    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:33.956761 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:33.956717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:36.787413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:36.787337    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 18 19:46:46.788071 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:46.788033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:46:48.924354 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:48.924323    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:46:48.924777 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:48.924568    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container" containerID="cri-o://6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b" gracePeriod=30
Sep 18 19:46:48.924777 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:48.924627    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kube-rbac-proxy" containerID="cri-o://21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325" gracePeriod=30
Sep 18 19:46:49.301728 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:49.301684    2583 generic.go:356] "Generic (PLEG): container finished" podID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerID="21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325" exitCode=2
Sep 18 19:46:49.301910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:49.301751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerDied","Data":"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"}
Sep 18 19:46:50.776937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:50.776897    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:46:50.777354 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:50.777167    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container" containerID="cri-o://200288bd275d3ec143fc1ca0a078505cd68c4c5a6feabc1377f252875e4fbf44" gracePeriod=30
Sep 18 19:46:50.777354 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:50.777200    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kube-rbac-proxy" containerID="cri-o://e7e19a8e0b226f2a2e7006d0504bb11175e9bbbec33dfb817ff12db22f938d11" gracePeriod=30
Sep 18 19:46:51.313134 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:51.313097    2583 generic.go:356] "Generic (PLEG): container finished" podID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerID="e7e19a8e0b226f2a2e7006d0504bb11175e9bbbec33dfb817ff12db22f938d11" exitCode=2
Sep 18 19:46:51.313299 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:51.313167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerDied","Data":"e7e19a8e0b226f2a2e7006d0504bb11175e9bbbec33dfb817ff12db22f938d11"}
Sep 18 19:46:51.469830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:51.469782    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 18 19:46:53.951179 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:53.951133    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 18 19:46:54.275149 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.275116    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:46:54.324571 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324537    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:46:54.324868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324855    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="storage-initializer"
Sep 18 19:46:54.324868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324868    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="storage-initializer"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324882    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kube-rbac-proxy"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324887    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kube-rbac-proxy"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324899    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324904    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324920    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324926    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324931    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324936    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324948    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="storage-initializer"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.324954    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="storage-initializer"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.325005    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kube-rbac-proxy"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.325014    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f4caf0-6585-4b3b-ab7f-3b3c3cc226a4" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.325021    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kserve-container"
Sep 18 19:46:54.325067 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.325028    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerName="kube-rbac-proxy"
Sep 18 19:46:54.327875 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.327836    2583 generic.go:356] "Generic (PLEG): container finished" podID="40858d80-4a8c-4061-a42c-79db3b2c9e67" containerID="6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b" exitCode=0
Sep 18 19:46:54.327980 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.327948    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"
Sep 18 19:46:54.328182 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.328162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerDied","Data":"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"}
Sep 18 19:46:54.328231 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.328217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s" event={"ID":"40858d80-4a8c-4061-a42c-79db3b2c9e67","Type":"ContainerDied","Data":"62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e"}
Sep 18 19:46:54.328271 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.328258    2583 scope.go:117] "RemoveContainer" containerID="21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"
Sep 18 19:46:54.328313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.328297    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.330928 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.330909    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:46:54.331288 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.331269    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:46:54.344701 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.344660    2583 scope.go:117] "RemoveContainer" containerID="6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"
Sep 18 19:46:54.346221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.346071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:46:54.355435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.355405    2583 scope.go:117] "RemoveContainer" containerID="cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"
Sep 18 19:46:54.364263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.364237    2583 scope.go:117] "RemoveContainer" containerID="21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"
Sep 18 19:46:54.364623 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:54.364601    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325\": container with ID starting with 21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325 not found: ID does not exist" containerID="21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"
Sep 18 19:46:54.364700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.364633    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325"} err="failed to get container status \"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325\": rpc error: code = NotFound desc = could not find container \"21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325\": container with ID starting with 21393d7d05fdf5446c090efdd0228e4394e46e8ac57b65f2472b574625b1a325 not found: ID does not exist"
Sep 18 19:46:54.364700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.364654    2583 scope.go:117] "RemoveContainer" containerID="6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"
Sep 18 19:46:54.364939 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:54.364910    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b\": container with ID starting with 6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b not found: ID does not exist" containerID="6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"
Sep 18 19:46:54.364982 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.364950    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b"} err="failed to get container status \"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b\": rpc error: code = NotFound desc = could not find container \"6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b\": container with ID starting with 6e851daaf44af9dc3078d055abd984f07534bf20ac46eb7b036d0cd78d6a2e5b not found: ID does not exist"
Sep 18 19:46:54.364982 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.364968    2583 scope.go:117] "RemoveContainer" containerID="cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"
Sep 18 19:46:54.365228 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:54.365208    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673\": container with ID starting with cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673 not found: ID does not exist" containerID="cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"
Sep 18 19:46:54.365286 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.365233    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673"} err="failed to get container status \"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673\": rpc error: code = NotFound desc = could not find container \"cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673\": container with ID starting with cdaa095e395ca0007c05db07e4c952a4a7d25b0a38bea6870a7b0eb376153673 not found: ID does not exist"
Sep 18 19:46:54.367743 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config\") pod \"40858d80-4a8c-4061-a42c-79db3b2c9e67\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") "
Sep 18 19:46:54.367792 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367777    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") pod \"40858d80-4a8c-4061-a42c-79db3b2c9e67\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") "
Sep 18 19:46:54.367870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g87\" (UniqueName: \"kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.367929 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.367989 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.367989 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.367983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.368097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.368011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.368097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.368023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40858d80-4a8c-4061-a42c-79db3b2c9e67" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:54.368097 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.368076    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.369778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.369759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40858d80-4a8c-4061-a42c-79db3b2c9e67" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:46:54.468553 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location\") pod \"40858d80-4a8c-4061-a42c-79db3b2c9e67\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") "
Sep 18 19:46:54.468553 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468557    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdw6g\" (UniqueName: \"kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g\") pod \"40858d80-4a8c-4061-a42c-79db3b2c9e67\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") "
Sep 18 19:46:54.468785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468620    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert\") pod \"40858d80-4a8c-4061-a42c-79db3b2c9e67\" (UID: \"40858d80-4a8c-4061-a42c-79db3b2c9e67\") "
Sep 18 19:46:54.468785 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.468862 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.468862 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.468961 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g87\" (UniqueName: \"kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.469017 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.469059 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.469041    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40858d80-4a8c-4061-a42c-79db3b2c9e67-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.469159 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.468922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40858d80-4a8c-4061-a42c-79db3b2c9e67" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:54.469440 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.469419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.469585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.469438    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40858d80-4a8c-4061-a42c-79db3b2c9e67" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:54.469708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.469682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.469779 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.469682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.471049 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.471017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g" (OuterVolumeSpecName: "kube-api-access-gdw6g") pod "40858d80-4a8c-4061-a42c-79db3b2c9e67" (UID: "40858d80-4a8c-4061-a42c-79db3b2c9e67"). InnerVolumeSpecName "kube-api-access-gdw6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:54.471382 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.471343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.478421 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.478399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g87\" (UniqueName: \"kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87\") pod \"isvc-canary-rollback-predictor-794fd55d6c-mfrhq\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.569610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.569525    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40858d80-4a8c-4061-a42c-79db3b2c9e67-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.569610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.569553    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdw6g\" (UniqueName: \"kubernetes.io/projected/40858d80-4a8c-4061-a42c-79db3b2c9e67-kube-api-access-gdw6g\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.569610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.569564    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40858d80-4a8c-4061-a42c-79db3b2c9e67-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:54.636047 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:54.635785    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40858d80_4a8c_4061_a42c_79db3b2c9e67.slice/crio-62c181500e1a558f2e9dda6b54f855d0b53356224c7329ff83220d70dc9ee59e\": RecentStats: unable to find data in memory cache]"
Sep 18 19:46:54.645204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.645178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:46:54.653707 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.653680    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:46:54.661169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.661145    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7d7f6d88c6-wfj8s"]
Sep 18 19:46:54.923787 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:54.923760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:46:54.926167 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:46:54.926132    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4e77bd_25ca_4d24_9e17_46ca0682f91f.slice/crio-107852b6a6e00db1217d8f1245f387d33629a5d80a1ea4baefca60094634538c WatchSource:0}: Error finding container 107852b6a6e00db1217d8f1245f387d33629a5d80a1ea4baefca60094634538c: Status 404 returned error can't find the container with id 107852b6a6e00db1217d8f1245f387d33629a5d80a1ea4baefca60094634538c
Sep 18 19:46:55.333931 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.333891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerStarted","Data":"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"}
Sep 18 19:46:55.334438 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.333972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerStarted","Data":"107852b6a6e00db1217d8f1245f387d33629a5d80a1ea4baefca60094634538c"}
Sep 18 19:46:55.791707 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.791672    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40858d80-4a8c-4061-a42c-79db3b2c9e67" path="/var/lib/kubelet/pods/40858d80-4a8c-4061-a42c-79db3b2c9e67/volumes"
Sep 18 19:46:55.911772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.911734    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:46:55.915528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.915508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:55.918344 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.918318    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 18 19:46:55.918486 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.918371    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 18 19:46:55.927547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.927515    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:46:55.981138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.981088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:55.981350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.981159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:55.981350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.981215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:55.981350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.981245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:55.981350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:55.981275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56vc\" (UniqueName: \"kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.081889 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.081795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.081889 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.081886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.081921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.081958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.081985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56vc\" (UniqueName: \"kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082117 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:56.082104    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 19:46:56.082326 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:46:56.082178    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls podName:15beaa70-4ac8-43f0-895a-6acabf7eb779 nodeName:}" failed. No retries permitted until 2026-09-18 19:46:56.582159083 +0000 UTC m=+1948.957585367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls") pod "isvc-tensorflow-predictor-6dfddc978b-9jtks" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 18 19:46:56.082433 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.082413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082581 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.082561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.082619 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.082599    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.090660 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.090623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56vc\" (UniqueName: \"kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.343302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.343214    2583 generic.go:356] "Generic (PLEG): container finished" podID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerID="200288bd275d3ec143fc1ca0a078505cd68c4c5a6feabc1377f252875e4fbf44" exitCode=0
Sep 18 19:46:56.343302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.343278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerDied","Data":"200288bd275d3ec143fc1ca0a078505cd68c4c5a6feabc1377f252875e4fbf44"}
Sep 18 19:46:56.435721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.435692    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:46:56.485515 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") pod \"157aa436-fd40-46cd-870f-b1c02bba5f39\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") "
Sep 18 19:46:56.485708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config\") pod \"157aa436-fd40-46cd-870f-b1c02bba5f39\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") "
Sep 18 19:46:56.485708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location\") pod \"157aa436-fd40-46cd-870f-b1c02bba5f39\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") "
Sep 18 19:46:56.485708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert\") pod \"157aa436-fd40-46cd-870f-b1c02bba5f39\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") "
Sep 18 19:46:56.485947 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "157aa436-fd40-46cd-870f-b1c02bba5f39" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:46:56.486008 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "157aa436-fd40-46cd-870f-b1c02bba5f39" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:56.486008 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.485969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "157aa436-fd40-46cd-870f-b1c02bba5f39" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:46:56.487673 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.487651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "157aa436-fd40-46cd-870f-b1c02bba5f39" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:46:56.586006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.585957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfj28\" (UniqueName: \"kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28\") pod \"157aa436-fd40-46cd-870f-b1c02bba5f39\" (UID: \"157aa436-fd40-46cd-870f-b1c02bba5f39\") "
Sep 18 19:46:56.586223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.586132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.586223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.586204    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/157aa436-fd40-46cd-870f-b1c02bba5f39-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:56.586223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.586215    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:56.586223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.586224    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/157aa436-fd40-46cd-870f-b1c02bba5f39-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:56.586471 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.586233    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/157aa436-fd40-46cd-870f-b1c02bba5f39-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:56.588070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.588043    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28" (OuterVolumeSpecName: "kube-api-access-xfj28") pod "157aa436-fd40-46cd-870f-b1c02bba5f39" (UID: "157aa436-fd40-46cd-870f-b1c02bba5f39"). InnerVolumeSpecName "kube-api-access-xfj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:46:56.588640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.588621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") pod \"isvc-tensorflow-predictor-6dfddc978b-9jtks\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:56.687005 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.686956    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfj28\" (UniqueName: \"kubernetes.io/projected/157aa436-fd40-46cd-870f-b1c02bba5f39-kube-api-access-xfj28\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:46:56.825908 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:56.825868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:46:57.102056 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.102018    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:46:57.104893 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:46:57.104862    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15beaa70_4ac8_43f0_895a_6acabf7eb779.slice/crio-d87dfa294acba671d55fb18ccf61c4e2b1737f0e1c58249b452e1ecd5c5fcec8 WatchSource:0}: Error finding container d87dfa294acba671d55fb18ccf61c4e2b1737f0e1c58249b452e1ecd5c5fcec8: Status 404 returned error can't find the container with id d87dfa294acba671d55fb18ccf61c4e2b1737f0e1c58249b452e1ecd5c5fcec8
Sep 18 19:46:57.350919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.350888    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"
Sep 18 19:46:57.350919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.350892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s" event={"ID":"157aa436-fd40-46cd-870f-b1c02bba5f39","Type":"ContainerDied","Data":"2ac29e27b30a4501376b12a577814f43f5a7596a03af1e01a793ed02a5f79fa4"}
Sep 18 19:46:57.351416 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.350981    2583 scope.go:117] "RemoveContainer" containerID="e7e19a8e0b226f2a2e7006d0504bb11175e9bbbec33dfb817ff12db22f938d11"
Sep 18 19:46:57.352844 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.352818    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerStarted","Data":"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"}
Sep 18 19:46:57.352985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.352881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerStarted","Data":"d87dfa294acba671d55fb18ccf61c4e2b1737f0e1c58249b452e1ecd5c5fcec8"}
Sep 18 19:46:57.363604 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.363584    2583 scope.go:117] "RemoveContainer" containerID="200288bd275d3ec143fc1ca0a078505cd68c4c5a6feabc1377f252875e4fbf44"
Sep 18 19:46:57.372176 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.372151    2583 scope.go:117] "RemoveContainer" containerID="3436df55843915a916976cd59b51b362289bca18a83f0c98e5255ce59db521e6"
Sep 18 19:46:57.396053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.395967    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:46:57.400673 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.400636    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-f7b4585f-7nz4s"]
Sep 18 19:46:57.792597 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:57.792562    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" path="/var/lib/kubelet/pods/157aa436-fd40-46cd-870f-b1c02bba5f39/volumes"
Sep 18 19:46:59.366954 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:59.366920    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerID="1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3" exitCode=0
Sep 18 19:46:59.367350 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:46:59.366994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerDied","Data":"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"}
Sep 18 19:47:00.374497 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:00.374464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerStarted","Data":"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"}
Sep 18 19:47:00.374906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:00.374528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerStarted","Data":"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"}
Sep 18 19:47:00.374906 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:00.374725    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:47:00.401246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:00.401177    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podStartSLOduration=2.648870372 podStartE2EDuration="6.401152703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752282331s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:00.399634907 +0000 UTC m=+1952.775061202" watchObservedRunningTime="2026-09-18 19:47:00.401152703 +0000 UTC m=+1952.776578997"
Sep 18 19:47:01.379854 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:01.379813    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:47:01.381457 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:01.381427    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:02.386133 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:02.386089    2583 generic.go:356] "Generic (PLEG): container finished" podID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerID="cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b" exitCode=0
Sep 18 19:47:02.386654 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:02.386163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerDied","Data":"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"}
Sep 18 19:47:02.386758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:02.386724    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:02.387794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:02.387775    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:47:06.413932 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.413897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerStarted","Data":"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"}
Sep 18 19:47:06.414447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.413961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerStarted","Data":"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"}
Sep 18 19:47:06.414447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.414241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:47:06.414447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.414391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:47:06.415278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.415254    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 19:47:06.436068 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:06.435963    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podStartSLOduration=3.151541132 podStartE2EDuration="11.435947336s" totalImagesPullingTime="3.779169858s" totalInitContainerRuntime="4.505236346s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:06.434683039 +0000 UTC m=+1958.810109360" watchObservedRunningTime="2026-09-18 19:47:06.435947336 +0000 UTC m=+1958.811373628"
Sep 18 19:47:07.391441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:07.391404    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:47:07.392165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:07.392129    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:07.418915 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:07.418870    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 19:47:12.423082 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:12.422995    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:47:12.423574 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:12.423551    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 18 19:47:17.392225 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:17.392179    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:22.424434 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:22.424398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:47:27.392759 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:27.392717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:37.392109 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:37.392072    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:37.639664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:37.639631    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:47:37.639921 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:37.639887    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container" containerID="cri-o://a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35" gracePeriod=30
Sep 18 19:47:37.639998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:37.639909    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" containerID="cri-o://7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd" gracePeriod=30
Sep 18 19:47:38.568863 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:38.568823    2583 generic.go:356] "Generic (PLEG): container finished" podID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerID="7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd" exitCode=2
Sep 18 19:47:38.569260 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:38.568888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerDied","Data":"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"}
Sep 18 19:47:42.419685 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:42.419638    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:47:47.392411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:47.392355    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:47.419593 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:47.419542    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:47:52.419735 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:52.419691    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:47:52.420151 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:52.419835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:47:57.392526 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:57.392478    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:47:57.419310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:47:57.419265    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:48:02.419320 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:02.419271    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:48:07.392261 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:07.392220    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:48:07.419532 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:07.419490    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 18 19:48:08.285076 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.285042    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:48:08.390671 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.390626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config\") pod \"15beaa70-4ac8-43f0-895a-6acabf7eb779\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") "
Sep 18 19:48:08.390671 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.390681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location\") pod \"15beaa70-4ac8-43f0-895a-6acabf7eb779\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") "
Sep 18 19:48:08.390933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.390699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") pod \"15beaa70-4ac8-43f0-895a-6acabf7eb779\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") "
Sep 18 19:48:08.390933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.390720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p56vc\" (UniqueName: \"kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc\") pod \"15beaa70-4ac8-43f0-895a-6acabf7eb779\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") "
Sep 18 19:48:08.390933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.390755    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert\") pod \"15beaa70-4ac8-43f0-895a-6acabf7eb779\" (UID: \"15beaa70-4ac8-43f0-895a-6acabf7eb779\") "
Sep 18 19:48:08.391158 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.391123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15beaa70-4ac8-43f0-895a-6acabf7eb779" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:08.391244 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.391215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15beaa70-4ac8-43f0-895a-6acabf7eb779" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:08.392958 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.392916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15beaa70-4ac8-43f0-895a-6acabf7eb779" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:08.393335 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.392930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc" (OuterVolumeSpecName: "kube-api-access-p56vc") pod "15beaa70-4ac8-43f0-895a-6acabf7eb779" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779"). InnerVolumeSpecName "kube-api-access-p56vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:08.402115 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.402071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15beaa70-4ac8-43f0-895a-6acabf7eb779" (UID: "15beaa70-4ac8-43f0-895a-6acabf7eb779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:08.491829 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.491789    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.491829 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.491825    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.492138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.491840    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15beaa70-4ac8-43f0-895a-6acabf7eb779-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.492138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.491852    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15beaa70-4ac8-43f0-895a-6acabf7eb779-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.492138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.491864    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56vc\" (UniqueName: \"kubernetes.io/projected/15beaa70-4ac8-43f0-895a-6acabf7eb779-kube-api-access-p56vc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:08.699584 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.699546    2583 generic.go:356] "Generic (PLEG): container finished" podID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerID="a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35" exitCode=137
Sep 18 19:48:08.699877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.699611    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"
Sep 18 19:48:08.699877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.699634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerDied","Data":"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"}
Sep 18 19:48:08.699877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.699735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks" event={"ID":"15beaa70-4ac8-43f0-895a-6acabf7eb779","Type":"ContainerDied","Data":"d87dfa294acba671d55fb18ccf61c4e2b1737f0e1c58249b452e1ecd5c5fcec8"}
Sep 18 19:48:08.699877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.699763    2583 scope.go:117] "RemoveContainer" containerID="7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"
Sep 18 19:48:08.709594 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.709574    2583 scope.go:117] "RemoveContainer" containerID="a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"
Sep 18 19:48:08.717888 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.717869    2583 scope.go:117] "RemoveContainer" containerID="cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"
Sep 18 19:48:08.725982 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.725961    2583 scope.go:117] "RemoveContainer" containerID="7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"
Sep 18 19:48:08.726313 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:08.726293    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd\": container with ID starting with 7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd not found: ID does not exist" containerID="7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"
Sep 18 19:48:08.726404 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.726328    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd"} err="failed to get container status \"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd\": rpc error: code = NotFound desc = could not find container \"7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd\": container with ID starting with 7a31b59790ea830bf415c57e1b536239f2b31fa3e932003b410f41a292a21bcd not found: ID does not exist"
Sep 18 19:48:08.726404 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.726354    2583 scope.go:117] "RemoveContainer" containerID="a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"
Sep 18 19:48:08.726680 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:08.726660    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35\": container with ID starting with a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35 not found: ID does not exist" containerID="a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"
Sep 18 19:48:08.726726 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.726688    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35"} err="failed to get container status \"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35\": rpc error: code = NotFound desc = could not find container \"a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35\": container with ID starting with a5a00c01f2a157ec3426ab471e8a1409b14a5a7bbbaed9b01f361d505452ce35 not found: ID does not exist"
Sep 18 19:48:08.726726 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.726704    2583 scope.go:117] "RemoveContainer" containerID="cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"
Sep 18 19:48:08.726969 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:08.726949    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b\": container with ID starting with cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b not found: ID does not exist" containerID="cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"
Sep 18 19:48:08.727022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.726976    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b"} err="failed to get container status \"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b\": rpc error: code = NotFound desc = could not find container \"cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b\": container with ID starting with cb04a8d4f6c52162f9568e898a4f4ccb4ed0bf0f66996c4b185eaada7eebd98b not found: ID does not exist"
Sep 18 19:48:08.737178 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.737145    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:48:08.739141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:08.739121    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6dfddc978b-9jtks"]
Sep 18 19:48:09.791897 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:09.791862    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" path="/var/lib/kubelet/pods/15beaa70-4ac8-43f0-895a-6acabf7eb779/volumes"
Sep 18 19:48:17.392900 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:17.392853    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 19:48:27.294983 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.294945    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:48:27.295512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.295171    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container" containerID="cri-o://dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428" gracePeriod=30
Sep 18 19:48:27.295512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.295275    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy" containerID="cri-o://790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3" gracePeriod=30
Sep 18 19:48:27.296263 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.296241    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:48:27.387336 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.387289    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 19:48:27.787099 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.787066    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerID="790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3" exitCode=2
Sep 18 19:48:27.796673 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:27.796638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerDied","Data":"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"}
Sep 18 19:48:29.348651 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348614    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348936    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="storage-initializer"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348948    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="storage-initializer"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348956    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="storage-initializer"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348962    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="storage-initializer"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348969    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348974    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348991    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.348997    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349004    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349011    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349019    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349040 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349024    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349072    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kserve-container"
Sep 18 19:48:29.349429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349080    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="15beaa70-4ac8-43f0-895a-6acabf7eb779" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349086    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kube-rbac-proxy"
Sep 18 19:48:29.349429 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.349092    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="157aa436-fd40-46cd-870f-b1c02bba5f39" containerName="kserve-container"
Sep 18 19:48:29.352325 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.352309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.355297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.355277    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 18 19:48:29.355749 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.355734    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 18 19:48:29.368892 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.368868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:48:29.485896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.485855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.485896 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.485900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.486117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.485934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdw6\" (UniqueName: \"kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.486117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.485977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.486117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.486016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.586851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.586799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.586851 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.586862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587073 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.586897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587073 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.586931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdw6\" (UniqueName: \"kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587073 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.586962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587234 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:29.587083    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 18 19:48:29.587234 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:29.587169    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls podName:3e623df9-af69-470b-a5bf-e22885bf4f4b nodeName:}" failed. No retries permitted until 2026-09-18 19:48:30.087147702 +0000 UTC m=+2042.462573974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls") pod "isvc-triton-predictor-7d578d8b99-5f5ng" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b") : secret "isvc-triton-predictor-serving-cert" not found
Sep 18 19:48:29.587315 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.587263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587719 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.587697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.587772 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.587697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:29.596963 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:29.596935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdw6\" (UniqueName: \"kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:30.091909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.091868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:30.094352 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.094324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") pod \"isvc-triton-predictor-7d578d8b99-5f5ng\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:30.262622 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.262584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:48:30.542648 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.542622    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:48:30.545193 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:48:30.545162    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e623df9_af69_470b_a5bf_e22885bf4f4b.slice/crio-a3c973fab77dd9849426d2440d0b8c8c6376e42fa26b4b28d1c88fe107592e1f WatchSource:0}: Error finding container a3c973fab77dd9849426d2440d0b8c8c6376e42fa26b4b28d1c88fe107592e1f: Status 404 returned error can't find the container with id a3c973fab77dd9849426d2440d0b8c8c6376e42fa26b4b28d1c88fe107592e1f
Sep 18 19:48:30.800743 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.800650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerStarted","Data":"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"}
Sep 18 19:48:30.800743 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:30.800720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerStarted","Data":"a3c973fab77dd9849426d2440d0b8c8c6376e42fa26b4b28d1c88fe107592e1f"}
Sep 18 19:48:32.442626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.442601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:48:32.612068 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612036    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls\") pod \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") "
Sep 18 19:48:32.612254 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config\") pod \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") "
Sep 18 19:48:32.612254 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2g87\" (UniqueName: \"kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87\") pod \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") "
Sep 18 19:48:32.612254 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location\") pod \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") "
Sep 18 19:48:32.612254 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612206    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert\") pod \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\" (UID: \"5c4e77bd-25ca-4d24-9e17-46ca0682f91f\") "
Sep 18 19:48:32.612612 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612587    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c4e77bd-25ca-4d24-9e17-46ca0682f91f" (UID: "5c4e77bd-25ca-4d24-9e17-46ca0682f91f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:48:32.612717 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612631    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c4e77bd-25ca-4d24-9e17-46ca0682f91f" (UID: "5c4e77bd-25ca-4d24-9e17-46ca0682f91f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:32.612717 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.612642    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c4e77bd-25ca-4d24-9e17-46ca0682f91f" (UID: "5c4e77bd-25ca-4d24-9e17-46ca0682f91f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:48:32.614348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.614322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87" (OuterVolumeSpecName: "kube-api-access-s2g87") pod "5c4e77bd-25ca-4d24-9e17-46ca0682f91f" (UID: "5c4e77bd-25ca-4d24-9e17-46ca0682f91f"). InnerVolumeSpecName "kube-api-access-s2g87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:48:32.614410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.614354    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c4e77bd-25ca-4d24-9e17-46ca0682f91f" (UID: "5c4e77bd-25ca-4d24-9e17-46ca0682f91f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:48:32.713427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.713388    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.713427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.713420    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.713427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.713429    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.713427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.713437    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g87\" (UniqueName: \"kubernetes.io/projected/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kube-api-access-s2g87\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.713684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.713446    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c4e77bd-25ca-4d24-9e17-46ca0682f91f-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:48:32.811149 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.811113    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerID="dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428" exitCode=0
Sep 18 19:48:32.811302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.811177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerDied","Data":"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"}
Sep 18 19:48:32.811302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.811187    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"
Sep 18 19:48:32.811302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.811237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" event={"ID":"5c4e77bd-25ca-4d24-9e17-46ca0682f91f","Type":"ContainerDied","Data":"107852b6a6e00db1217d8f1245f387d33629a5d80a1ea4baefca60094634538c"}
Sep 18 19:48:32.811302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.811272    2583 scope.go:117] "RemoveContainer" containerID="790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"
Sep 18 19:48:32.821838 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.821714    2583 scope.go:117] "RemoveContainer" containerID="dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"
Sep 18 19:48:32.829477 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.829449    2583 scope.go:117] "RemoveContainer" containerID="1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"
Sep 18 19:48:32.841948 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.841926    2583 scope.go:117] "RemoveContainer" containerID="790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"
Sep 18 19:48:32.842287 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:32.842264    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3\": container with ID starting with 790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3 not found: ID does not exist" containerID="790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"
Sep 18 19:48:32.842396 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.842294    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3"} err="failed to get container status \"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3\": rpc error: code = NotFound desc = could not find container \"790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3\": container with ID starting with 790e379c67354110c059fe95574733255b9b95f2b349ae395fb25422fc9a09c3 not found: ID does not exist"
Sep 18 19:48:32.842396 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.842312    2583 scope.go:117] "RemoveContainer" containerID="dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"
Sep 18 19:48:32.842640 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:32.842618    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428\": container with ID starting with dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428 not found: ID does not exist" containerID="dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"
Sep 18 19:48:32.842691 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.842657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428"} err="failed to get container status \"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428\": rpc error: code = NotFound desc = could not find container \"dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428\": container with ID starting with dd3722cbac821bc6a1ba3bbe71e78341d10e79c66507551c23a5829d34631428 not found: ID does not exist"
Sep 18 19:48:32.842691 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.842680    2583 scope.go:117] "RemoveContainer" containerID="1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"
Sep 18 19:48:32.843036 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:48:32.842990    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3\": container with ID starting with 1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3 not found: ID does not exist" containerID="1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"
Sep 18 19:48:32.843036 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.843022    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3"} err="failed to get container status \"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3\": rpc error: code = NotFound desc = could not find container \"1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3\": container with ID starting with 1ccea937207de483c84c603df2046f15da47aaa8eec6c87efdade52c455659c3 not found: ID does not exist"
Sep 18 19:48:32.845006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.844988    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:48:32.849651 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:32.849630    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq"]
Sep 18 19:48:33.387626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:33.387573    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-mfrhq" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": context deadline exceeded"
Sep 18 19:48:33.791125 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:33.791093    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" path="/var/lib/kubelet/pods/5c4e77bd-25ca-4d24-9e17-46ca0682f91f/volumes"
Sep 18 19:48:34.241852 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.241816    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:48:34.242204 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242187    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container"
Sep 18 19:48:34.242306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242207    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container"
Sep 18 19:48:34.242306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242238    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy"
Sep 18 19:48:34.242306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242246    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy"
Sep 18 19:48:34.242306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242266    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="storage-initializer"
Sep 18 19:48:34.242306 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242274    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="storage-initializer"
Sep 18 19:48:34.242593 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242430    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kserve-container"
Sep 18 19:48:34.242593 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.242444    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4e77bd-25ca-4d24-9e17-46ca0682f91f" containerName="kube-rbac-proxy"
Sep 18 19:48:34.247031 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.247009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.249861 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.249836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:48:34.250221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.250200    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:48:34.261245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.261219    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:48:34.327101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.327064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9p9\" (UniqueName: \"kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.327101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.327103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.327333 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.327183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.327333 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.327230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.327333 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.327271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.428737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.428700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.428737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.428749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.428924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.428982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9p9\" (UniqueName: \"kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.429003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.429082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.429602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.429748 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.429610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.431314 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.431291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.438585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.438561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9p9\" (UniqueName: \"kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9\") pod \"isvc-canary-rollback-predictor-794fd55d6c-pdpsf\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.558102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.558002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:34.823282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.823197    2583 generic.go:356] "Generic (PLEG): container finished" podID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerID="f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff" exitCode=0
Sep 18 19:48:34.823282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.823273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerDied","Data":"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"}
Sep 18 19:48:34.838080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:34.838057    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:48:34.840048 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:48:34.840025    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf7d8bb_ed1f_42fb_b60a_359d0082402e.slice/crio-55676bec9b3dd5e1051d068a6db38bdebd9825078994236c28704f2ae114a194 WatchSource:0}: Error finding container 55676bec9b3dd5e1051d068a6db38bdebd9825078994236c28704f2ae114a194: Status 404 returned error can't find the container with id 55676bec9b3dd5e1051d068a6db38bdebd9825078994236c28704f2ae114a194
Sep 18 19:48:35.833389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:35.832903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerStarted","Data":"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"}
Sep 18 19:48:35.833389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:35.832997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerStarted","Data":"55676bec9b3dd5e1051d068a6db38bdebd9825078994236c28704f2ae114a194"}
Sep 18 19:48:38.872267 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:38.870974    2583 generic.go:356] "Generic (PLEG): container finished" podID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerID="301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82" exitCode=0
Sep 18 19:48:38.872267 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:38.871106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerDied","Data":"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"}
Sep 18 19:48:39.891737 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.891701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerStarted","Data":"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"}
Sep 18 19:48:39.892305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.891778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerStarted","Data":"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"}
Sep 18 19:48:39.893140 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.893116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:39.893258 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.893170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:39.894572 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.894544    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:48:39.924782 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:39.924719    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podStartSLOduration=2.045852351 podStartE2EDuration="5.924698196s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.878845845s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:39.92377015 +0000 UTC m=+2052.299196444" watchObservedRunningTime="2026-09-18 19:48:39.924698196 +0000 UTC m=+2052.300124484"
Sep 18 19:48:40.900721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:40.899218    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:48:45.906925 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:45.906875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:48:45.908029 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:45.907967    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:48:55.908215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:48:55.908144    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:05.909175 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:05.909022    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:15.908169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:15.908088    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:25.908246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:25.908198    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:35.908647 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:35.908599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:45.908627 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:45.908565    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:49:55.908786 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:49:55.908742    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:50:03.787158 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:03.787114    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:50:13.787688 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:13.787576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 18 19:50:23.792732 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:23.792700    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:50:24.611273 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:24.611233    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:50:24.611668 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:24.611518    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container" containerID="cri-o://08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10" gracePeriod=30
Sep 18 19:50:24.611668 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:24.611555    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kube-rbac-proxy" containerID="cri-o://86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba" gracePeriod=30
Sep 18 19:50:25.899329 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:25.899282    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 18 19:50:29.187055 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:29.187016    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:50:29.187055 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:29.187042    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:50:29.486868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:29.486785    2583 generic.go:356] "Generic (PLEG): container finished" podID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerID="86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba" exitCode=2
Sep 18 19:50:29.487013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:29.486862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerDied","Data":"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"}
Sep 18 19:50:30.026450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.026412    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:50:30.048909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.048881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.052391 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.052343    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:50:30.052757 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.052738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:50:30.052833 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.052756    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:50:30.116451 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.116413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.116588 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.116465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.116588 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.116524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.116670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.116605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfzc\" (UniqueName: \"kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.116670 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.116651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.217582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.217546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.217607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfzc\" (UniqueName: \"kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.217636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.217742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.217775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.218073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218528 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.218508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.218750 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.218726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.221313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.221290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.295864 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.295834    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:50:30.325686 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.325654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfzc\" (UniqueName: \"kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc\") pod \"isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.361708 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.361673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:30.420174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420146    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config\") pod \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") "
Sep 18 19:50:30.420310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420220    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location\") pod \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") "
Sep 18 19:50:30.420473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls\") pod \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") "
Sep 18 19:50:30.420473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert\") pod \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") "
Sep 18 19:50:30.420585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420542    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cf7d8bb-ed1f-42fb-b60a-359d0082402e" (UID: "4cf7d8bb-ed1f-42fb-b60a-359d0082402e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:50:30.420677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj9p9\" (UniqueName: \"kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9\") pod \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\" (UID: \"4cf7d8bb-ed1f-42fb-b60a-359d0082402e\") "
Sep 18 19:50:30.420902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cf7d8bb-ed1f-42fb-b60a-359d0082402e" (UID: "4cf7d8bb-ed1f-42fb-b60a-359d0082402e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:30.420988 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cf7d8bb-ed1f-42fb-b60a-359d0082402e" (UID: "4cf7d8bb-ed1f-42fb-b60a-359d0082402e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:30.421047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.420997    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.421047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.421019    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.423218 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.423170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9" (OuterVolumeSpecName: "kube-api-access-qj9p9") pod "4cf7d8bb-ed1f-42fb-b60a-359d0082402e" (UID: "4cf7d8bb-ed1f-42fb-b60a-359d0082402e"). InnerVolumeSpecName "kube-api-access-qj9p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:30.423419 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.423383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cf7d8bb-ed1f-42fb-b60a-359d0082402e" (UID: "4cf7d8bb-ed1f-42fb-b60a-359d0082402e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:50:30.493955 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.493911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerStarted","Data":"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"}
Sep 18 19:50:30.494139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.493992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerStarted","Data":"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"}
Sep 18 19:50:30.494139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.494030    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:30.495895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.495869    2583 generic.go:356] "Generic (PLEG): container finished" podID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerID="08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10" exitCode=0
Sep 18 19:50:30.496013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.495905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerDied","Data":"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"}
Sep 18 19:50:30.496013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.495925    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"
Sep 18 19:50:30.496013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.495963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf" event={"ID":"4cf7d8bb-ed1f-42fb-b60a-359d0082402e","Type":"ContainerDied","Data":"55676bec9b3dd5e1051d068a6db38bdebd9825078994236c28704f2ae114a194"}
Sep 18 19:50:30.496013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.495994    2583 scope.go:117] "RemoveContainer" containerID="86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"
Sep 18 19:50:30.505951 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.505859    2583 scope.go:117] "RemoveContainer" containerID="08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"
Sep 18 19:50:30.515277 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.515254    2583 scope.go:117] "RemoveContainer" containerID="301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"
Sep 18 19:50:30.522125 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.522088    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.522125 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.522127    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.522327 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.522142    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9p9\" (UniqueName: \"kubernetes.io/projected/4cf7d8bb-ed1f-42fb-b60a-359d0082402e-kube-api-access-qj9p9\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:30.524058 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.524039    2583 scope.go:117] "RemoveContainer" containerID="86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"
Sep 18 19:50:30.524484 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:30.524455    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba\": container with ID starting with 86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba not found: ID does not exist" containerID="86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"
Sep 18 19:50:30.524831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.524805    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba"} err="failed to get container status \"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba\": rpc error: code = NotFound desc = could not find container \"86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba\": container with ID starting with 86bca54f65c8bcbc9a8d4216f3990c3006b3aec2b104f3d87864412d57153dba not found: ID does not exist"
Sep 18 19:50:30.524917 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.524836    2583 scope.go:117] "RemoveContainer" containerID="08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"
Sep 18 19:50:30.525152 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:30.525136    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10\": container with ID starting with 08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10 not found: ID does not exist" containerID="08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"
Sep 18 19:50:30.525221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.525161    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10"} err="failed to get container status \"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10\": rpc error: code = NotFound desc = could not find container \"08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10\": container with ID starting with 08c5d6b7565346072cdad53d496c52cb8d6a4e4a05350203934e80b3c255fa10 not found: ID does not exist"
Sep 18 19:50:30.525221 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.525183    2583 scope.go:117] "RemoveContainer" containerID="301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"
Sep 18 19:50:30.525498 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:30.525479    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82\": container with ID starting with 301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82 not found: ID does not exist" containerID="301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"
Sep 18 19:50:30.525555 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.525504    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82"} err="failed to get container status \"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82\": rpc error: code = NotFound desc = could not find container \"301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82\": container with ID starting with 301eda98dc5ad097ef59af6f7fcd6f9895f581e2c5701046617db19ab3efec82 not found: ID does not exist"
Sep 18 19:50:30.543136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.543035    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podStartSLOduration=2.288281221 podStartE2EDuration="2m1.543019149s" totalImagesPullingTime="1m55.306786123s" totalInitContainerRuntime="3.947951805s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:30.541944867 +0000 UTC m=+2162.917371163" watchObservedRunningTime="2026-09-18 19:50:30.543019149 +0000 UTC m=+2162.918445442"
Sep 18 19:50:30.580102 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.580070    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:50:30.592326 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.592278    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-794fd55d6c-pdpsf"]
Sep 18 19:50:30.687890 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:30.687861    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:50:30.690110 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:50:30.690089    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840281db_bb3d_4958_a547_b93a1fe8ab0e.slice/crio-77246d9915c47315cf8aeb47c06cf51a2ae3f7fcdbfe26e41e51bb0593c24fc7 WatchSource:0}: Error finding container 77246d9915c47315cf8aeb47c06cf51a2ae3f7fcdbfe26e41e51bb0593c24fc7: Status 404 returned error can't find the container with id 77246d9915c47315cf8aeb47c06cf51a2ae3f7fcdbfe26e41e51bb0593c24fc7
Sep 18 19:50:31.503760 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:31.503720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerStarted","Data":"04c59c7d4c56be64c26e0125258f9934dbee913f08b7e9c43a2b0119a3d0f169"}
Sep 18 19:50:31.504240 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:31.503792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerStarted","Data":"77246d9915c47315cf8aeb47c06cf51a2ae3f7fcdbfe26e41e51bb0593c24fc7"}
Sep 18 19:50:31.504240 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:31.504186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:31.505402 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:31.505375    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 19:50:31.792060 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:31.791977    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" path="/var/lib/kubelet/pods/4cf7d8bb-ed1f-42fb-b60a-359d0082402e/volumes"
Sep 18 19:50:32.509038 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:32.508999    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 18 19:50:35.525192 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:35.525161    2583 generic.go:356] "Generic (PLEG): container finished" podID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerID="04c59c7d4c56be64c26e0125258f9934dbee913f08b7e9c43a2b0119a3d0f169" exitCode=0
Sep 18 19:50:35.525823 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:35.525218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerDied","Data":"04c59c7d4c56be64c26e0125258f9934dbee913f08b7e9c43a2b0119a3d0f169"}
Sep 18 19:50:36.531450 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.531415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerStarted","Data":"53a99b1312dcc3753f31f65135f560679878d59b01ffcf5c72ea5312f2472592"}
Sep 18 19:50:36.531812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.531477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerStarted","Data":"f5c956c747fa39094ab384001ec8a24b8d5981f54af91e61f258978fc588afa0"}
Sep 18 19:50:36.531812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.531734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:36.531904 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.531850    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:36.532978 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.532945    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:50:36.560868 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:36.560811    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podStartSLOduration=3.416442528 podStartE2EDuration="7.560792694s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.144350166s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:36.558224921 +0000 UTC m=+2168.933651213" watchObservedRunningTime="2026-09-18 19:50:36.560792694 +0000 UTC m=+2168.936218987"
Sep 18 19:50:37.513476 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:37.513445    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:37.514215 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:37.514195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:37.536228 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:37.536195    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:50:41.848285 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:41.848251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:50:41.848845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:41.848489    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container" containerID="cri-o://a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b" gracePeriod=30
Sep 18 19:50:41.848845 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:41.848516    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kube-rbac-proxy" containerID="cri-o://f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf" gracePeriod=30
Sep 18 19:50:42.509253 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:42.509214    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 18 19:50:42.539970 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:42.539939    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:50:42.540444 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:42.540416    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:50:42.560242 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:42.560212    2583 generic.go:356] "Generic (PLEG): container finished" podID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerID="f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf" exitCode=2
Sep 18 19:50:42.560407 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:42.560285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerDied","Data":"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"}
Sep 18 19:50:44.513373 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.513322    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:44.572468 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.572380    2583 generic.go:356] "Generic (PLEG): container finished" podID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerID="a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b" exitCode=0
Sep 18 19:50:44.572468 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.572450    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"
Sep 18 19:50:44.572468 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.572455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerDied","Data":"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"}
Sep 18 19:50:44.572680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.572520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng" event={"ID":"3e623df9-af69-470b-a5bf-e22885bf4f4b","Type":"ContainerDied","Data":"a3c973fab77dd9849426d2440d0b8c8c6376e42fa26b4b28d1c88fe107592e1f"}
Sep 18 19:50:44.572680 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.572546    2583 scope.go:117] "RemoveContainer" containerID="f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"
Sep 18 19:50:44.580704 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.580687    2583 scope.go:117] "RemoveContainer" containerID="a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"
Sep 18 19:50:44.588659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.588641    2583 scope.go:117] "RemoveContainer" containerID="f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"
Sep 18 19:50:44.596044 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596024    2583 scope.go:117] "RemoveContainer" containerID="f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"
Sep 18 19:50:44.596344 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:44.596324    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf\": container with ID starting with f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf not found: ID does not exist" containerID="f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"
Sep 18 19:50:44.596419 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596354    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf"} err="failed to get container status \"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf\": rpc error: code = NotFound desc = could not find container \"f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf\": container with ID starting with f49587c9914f72d4565e2218fa08851ad31bb35e81ca97700e8d20b698453cbf not found: ID does not exist"
Sep 18 19:50:44.596419 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596386    2583 scope.go:117] "RemoveContainer" containerID="a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"
Sep 18 19:50:44.596662 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:44.596643    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b\": container with ID starting with a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b not found: ID does not exist" containerID="a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"
Sep 18 19:50:44.596718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b"} err="failed to get container status \"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b\": rpc error: code = NotFound desc = could not find container \"a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b\": container with ID starting with a50f79244caf69d9155d1d8b53afc5f82b946084a9a26d11cb42441c7b1e780b not found: ID does not exist"
Sep 18 19:50:44.596718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596686    2583 scope.go:117] "RemoveContainer" containerID="f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"
Sep 18 19:50:44.596926 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:44.596902    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff\": container with ID starting with f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff not found: ID does not exist" containerID="f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"
Sep 18 19:50:44.597034 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.596929    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff"} err="failed to get container status \"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff\": rpc error: code = NotFound desc = could not find container \"f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff\": container with ID starting with f46e75f81db041c514a643cef8a9d50790633efbb839cdd191af2e75b49180ff not found: ID does not exist"
Sep 18 19:50:44.648585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.648550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfdw6\" (UniqueName: \"kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6\") pod \"3e623df9-af69-470b-a5bf-e22885bf4f4b\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") "
Sep 18 19:50:44.648790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.648612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config\") pod \"3e623df9-af69-470b-a5bf-e22885bf4f4b\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") "
Sep 18 19:50:44.648790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.648641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") pod \"3e623df9-af69-470b-a5bf-e22885bf4f4b\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") "
Sep 18 19:50:44.648790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.648723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert\") pod \"3e623df9-af69-470b-a5bf-e22885bf4f4b\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") "
Sep 18 19:50:44.648790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.648758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location\") pod \"3e623df9-af69-470b-a5bf-e22885bf4f4b\" (UID: \"3e623df9-af69-470b-a5bf-e22885bf4f4b\") "
Sep 18 19:50:44.649151 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.649121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e623df9-af69-470b-a5bf-e22885bf4f4b" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:44.649271 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.649140    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e623df9-af69-470b-a5bf-e22885bf4f4b" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:50:44.649397 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.649309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e623df9-af69-470b-a5bf-e22885bf4f4b" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:50:44.650820 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.650793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e623df9-af69-470b-a5bf-e22885bf4f4b" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:50:44.650918 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.650901    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6" (OuterVolumeSpecName: "kube-api-access-dfdw6") pod "3e623df9-af69-470b-a5bf-e22885bf4f4b" (UID: "3e623df9-af69-470b-a5bf-e22885bf4f4b"). InnerVolumeSpecName "kube-api-access-dfdw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:50:44.750406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.750347    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.750406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.750399    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e623df9-af69-470b-a5bf-e22885bf4f4b-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.750406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.750409    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e623df9-af69-470b-a5bf-e22885bf4f4b-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.750406 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.750419    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e623df9-af69-470b-a5bf-e22885bf4f4b-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.750721 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.750427    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdw6\" (UniqueName: \"kubernetes.io/projected/3e623df9-af69-470b-a5bf-e22885bf4f4b-kube-api-access-dfdw6\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:50:44.901084 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.901046    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:50:44.909065 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:44.909034    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-7d578d8b99-5f5ng"]
Sep 18 19:50:45.792571 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:45.792539    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" path="/var/lib/kubelet/pods/3e623df9-af69-470b-a5bf-e22885bf4f4b/volumes"
Sep 18 19:50:47.112255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112222    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112542    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="storage-initializer"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112554    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="storage-initializer"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112568    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="storage-initializer"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112574    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="storage-initializer"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112584    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112590    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112598    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kube-rbac-proxy"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112603    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kube-rbac-proxy"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112620    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kube-rbac-proxy"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112627    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kube-rbac-proxy"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112634    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112639    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container"
Sep 18 19:50:47.112683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112693    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kserve-container"
Sep 18 19:50:47.113146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112702    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e623df9-af69-470b-a5bf-e22885bf4f4b" containerName="kube-rbac-proxy"
Sep 18 19:50:47.113146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112708    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kube-rbac-proxy"
Sep 18 19:50:47.113146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.112716    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf7d8bb-ed1f-42fb-b60a-359d0082402e" containerName="kserve-container"
Sep 18 19:50:47.153568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.153536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.157472 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.157443    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:50:47.159373 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.159344    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 18 19:50:47.159711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.159691    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 18 19:50:47.270216 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.270174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.270427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.270232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.270427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.270288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbg7k\" (UniqueName: \"kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.270427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.270338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.270427 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.270413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371304 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371304 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371304 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371652 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbg7k\" (UniqueName: \"kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371652 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.371652 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:47.371404    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 18 19:50:47.371652 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:50:47.371475    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls podName:c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc nodeName:}" failed. No retries permitted until 2026-09-18 19:50:47.871453539 +0000 UTC m=+2180.246880043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls") pod "isvc-xgboost-predictor-9fb66b6c4-cwkm2" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 18 19:50:47.371902 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.371759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.372035 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.372015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.372149 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.372129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.386929 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.386896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbg7k\" (UniqueName: \"kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.875187 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.875149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:47.877630 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:47.877594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") pod \"isvc-xgboost-predictor-9fb66b6c4-cwkm2\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:48.065591 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:48.065554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:50:48.415517 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:48.415492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:50:48.417407 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:50:48.417379    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a0a316_004d_4c64_b2f0_8da6f6f5ebbc.slice/crio-07497099b7e025879ae2ac380da492eaddc0d238a336d09f53eb0d2fb306d9c9 WatchSource:0}: Error finding container 07497099b7e025879ae2ac380da492eaddc0d238a336d09f53eb0d2fb306d9c9: Status 404 returned error can't find the container with id 07497099b7e025879ae2ac380da492eaddc0d238a336d09f53eb0d2fb306d9c9
Sep 18 19:50:48.590839 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:48.590801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerStarted","Data":"07497099b7e025879ae2ac380da492eaddc0d238a336d09f53eb0d2fb306d9c9"}
Sep 18 19:50:49.596301 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:49.596260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerStarted","Data":"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"}
Sep 18 19:50:52.540426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:52.540382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:50:52.610032 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:52.609998    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerID="6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967" exitCode=0
Sep 18 19:50:52.610245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:50:52.610074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerDied","Data":"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"}
Sep 18 19:51:02.541441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:02.541397    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:11.709317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:11.709221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerStarted","Data":"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"}
Sep 18 19:51:11.709317 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:11.709296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerStarted","Data":"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"}
Sep 18 19:51:11.709865 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:11.709541    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:51:11.732266 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:11.732203    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podStartSLOduration=2.117842949 podStartE2EDuration="24.732187669s" totalImagesPullingTime="18.751494413s" totalInitContainerRuntime="3.862850307s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:51:11.731917208 +0000 UTC m=+2204.107343508" watchObservedRunningTime="2026-09-18 19:51:11.732187669 +0000 UTC m=+2204.107613961"
Sep 18 19:51:12.540858 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:12.540819    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:12.714197 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:12.714156    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:51:12.715442 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:12.715413    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:13.718963 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:13.718924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:18.724174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:18.724142    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:51:18.724746 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:18.724695    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:22.541084 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:22.541042    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:28.725518 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:28.725472    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:32.540880 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:32.540837    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:38.724643 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:38.724597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:42.540435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:42.540325    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:48.724778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:48.724731    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:51:52.540873 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:52.540831    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:51:58.725418 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:51:58.725357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:52:02.541541 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:02.541509    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:52:08.724996 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:08.724934    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:52:09.667387 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:09.667327    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:52:09.667630 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:09.667602    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" containerID="cri-o://f5c956c747fa39094ab384001ec8a24b8d5981f54af91e61f258978fc588afa0" gracePeriod=30
Sep 18 19:52:09.667711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:09.667680    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kube-rbac-proxy" containerID="cri-o://53a99b1312dcc3753f31f65135f560679878d59b01ffcf5c72ea5312f2472592" gracePeriod=30
Sep 18 19:52:09.978444 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:09.978318    2583 generic.go:356] "Generic (PLEG): container finished" podID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerID="53a99b1312dcc3753f31f65135f560679878d59b01ffcf5c72ea5312f2472592" exitCode=2
Sep 18 19:52:09.978816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:09.978399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerDied","Data":"53a99b1312dcc3753f31f65135f560679878d59b01ffcf5c72ea5312f2472592"}
Sep 18 19:52:12.536891 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:12.536839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 18 19:52:12.540606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:12.540576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 18 19:52:15.009600 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.009562    2583 generic.go:356] "Generic (PLEG): container finished" podID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerID="f5c956c747fa39094ab384001ec8a24b8d5981f54af91e61f258978fc588afa0" exitCode=0
Sep 18 19:52:15.009983 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.009633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerDied","Data":"f5c956c747fa39094ab384001ec8a24b8d5981f54af91e61f258978fc588afa0"}
Sep 18 19:52:15.113104 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.113075    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:52:15.247555 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247522    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls\") pod \"840281db-bb3d-4958-a547-b93a1fe8ab0e\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") "
Sep 18 19:52:15.247758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpfzc\" (UniqueName: \"kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc\") pod \"840281db-bb3d-4958-a547-b93a1fe8ab0e\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") "
Sep 18 19:52:15.247758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247635    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config\") pod \"840281db-bb3d-4958-a547-b93a1fe8ab0e\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") "
Sep 18 19:52:15.247758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert\") pod \"840281db-bb3d-4958-a547-b93a1fe8ab0e\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") "
Sep 18 19:52:15.247758 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247722    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location\") pod \"840281db-bb3d-4958-a547-b93a1fe8ab0e\" (UID: \"840281db-bb3d-4958-a547-b93a1fe8ab0e\") "
Sep 18 19:52:15.247993 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.247966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "840281db-bb3d-4958-a547-b93a1fe8ab0e" (UID: "840281db-bb3d-4958-a547-b93a1fe8ab0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:15.248089 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.248067    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "840281db-bb3d-4958-a547-b93a1fe8ab0e" (UID: "840281db-bb3d-4958-a547-b93a1fe8ab0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:15.248138 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.248105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "840281db-bb3d-4958-a547-b93a1fe8ab0e" (UID: "840281db-bb3d-4958-a547-b93a1fe8ab0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:15.249723 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.249696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "840281db-bb3d-4958-a547-b93a1fe8ab0e" (UID: "840281db-bb3d-4958-a547-b93a1fe8ab0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:15.249840 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.249801    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc" (OuterVolumeSpecName: "kube-api-access-xpfzc") pod "840281db-bb3d-4958-a547-b93a1fe8ab0e" (UID: "840281db-bb3d-4958-a547-b93a1fe8ab0e"). InnerVolumeSpecName "kube-api-access-xpfzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:15.348582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.348527    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/840281db-bb3d-4958-a547-b93a1fe8ab0e-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:15.348582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.348575    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/840281db-bb3d-4958-a547-b93a1fe8ab0e-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:15.348582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.348585    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfzc\" (UniqueName: \"kubernetes.io/projected/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-api-access-xpfzc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:15.348582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.348595    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:15.348582 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:15.348603    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/840281db-bb3d-4958-a547-b93a1fe8ab0e-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:16.015767 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.015736    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"
Sep 18 19:52:16.015767 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.015744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr" event={"ID":"840281db-bb3d-4958-a547-b93a1fe8ab0e","Type":"ContainerDied","Data":"77246d9915c47315cf8aeb47c06cf51a2ae3f7fcdbfe26e41e51bb0593c24fc7"}
Sep 18 19:52:16.016283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.015817    2583 scope.go:117] "RemoveContainer" containerID="53a99b1312dcc3753f31f65135f560679878d59b01ffcf5c72ea5312f2472592"
Sep 18 19:52:16.024222 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.024201    2583 scope.go:117] "RemoveContainer" containerID="f5c956c747fa39094ab384001ec8a24b8d5981f54af91e61f258978fc588afa0"
Sep 18 19:52:16.031665 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.031644    2583 scope.go:117] "RemoveContainer" containerID="04c59c7d4c56be64c26e0125258f9934dbee913f08b7e9c43a2b0119a3d0f169"
Sep 18 19:52:16.044006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.043980    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:52:16.050191 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:16.050135    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6fc7d5b8c9-hmzhr"]
Sep 18 19:52:17.795291 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:17.795256    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" path="/var/lib/kubelet/pods/840281db-bb3d-4958-a547-b93a1fe8ab0e/volumes"
Sep 18 19:52:18.724638 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:18.724595    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:52:28.725345 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:28.725312    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:52:37.557565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:37.557531    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:52:37.557942 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:37.557814    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" containerID="cri-o://bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059" gracePeriod=30
Sep 18 19:52:37.557995 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:37.557897    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kube-rbac-proxy" containerID="cri-o://a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e" gracePeriod=30
Sep 18 19:52:38.118466 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:38.118413    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerID="a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e" exitCode=2
Sep 18 19:52:38.118651 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:38.118487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerDied","Data":"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"}
Sep 18 19:52:38.719173 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:38.719133    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 18 19:52:38.725568 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:38.725540    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 18 19:52:41.800346 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.800320    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:52:41.865246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865154    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbg7k\" (UniqueName: \"kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k\") pod \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") "
Sep 18 19:52:41.865246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config\") pod \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") "
Sep 18 19:52:41.865246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location\") pod \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") "
Sep 18 19:52:41.865557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865262    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert\") pod \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") "
Sep 18 19:52:41.865557 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865288    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") pod \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\" (UID: \"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc\") "
Sep 18 19:52:41.865674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:41.865674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865647    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:41.865674 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.865646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:41.867469 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.867443    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k" (OuterVolumeSpecName: "kube-api-access-kbg7k") pod "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc"). InnerVolumeSpecName "kube-api-access-kbg7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:41.867575 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.867550    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" (UID: "c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:41.966172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.966130    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbg7k\" (UniqueName: \"kubernetes.io/projected/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-api-access-kbg7k\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:41.966172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.966165    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:41.966172 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.966178    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:41.966493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.966191    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:41.966493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:41.966203    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:52:42.136381 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.136276    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerID="bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059" exitCode=0
Sep 18 19:52:42.136381 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.136336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"
Sep 18 19:52:42.136381 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.136340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerDied","Data":"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"}
Sep 18 19:52:42.136616 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.136405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2" event={"ID":"c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc","Type":"ContainerDied","Data":"07497099b7e025879ae2ac380da492eaddc0d238a336d09f53eb0d2fb306d9c9"}
Sep 18 19:52:42.136616 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.136433    2583 scope.go:117] "RemoveContainer" containerID="a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"
Sep 18 19:52:42.147449 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.147430    2583 scope.go:117] "RemoveContainer" containerID="bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"
Sep 18 19:52:42.155451 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.155432    2583 scope.go:117] "RemoveContainer" containerID="6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"
Sep 18 19:52:42.162881 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.162825    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:52:42.163951 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.163931    2583 scope.go:117] "RemoveContainer" containerID="a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"
Sep 18 19:52:42.164322 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:52:42.164301    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e\": container with ID starting with a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e not found: ID does not exist" containerID="a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"
Sep 18 19:52:42.164415 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164333    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e"} err="failed to get container status \"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e\": rpc error: code = NotFound desc = could not find container \"a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e\": container with ID starting with a4f23c39bafb5ebb497b52de0791d7eae26e5d1101f69a14b0ecea32d3fbaf3e not found: ID does not exist"
Sep 18 19:52:42.164415 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164353    2583 scope.go:117] "RemoveContainer" containerID="bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"
Sep 18 19:52:42.164652 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:52:42.164635    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059\": container with ID starting with bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059 not found: ID does not exist" containerID="bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"
Sep 18 19:52:42.164717 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164662    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059"} err="failed to get container status \"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059\": rpc error: code = NotFound desc = could not find container \"bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059\": container with ID starting with bdcf777f205f0430f09841e75b35848e19e7ca90b6d5cb5be5cf7b7605bb3059 not found: ID does not exist"
Sep 18 19:52:42.164717 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164685    2583 scope.go:117] "RemoveContainer" containerID="6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"
Sep 18 19:52:42.164887 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164859    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-9fb66b6c4-cwkm2"]
Sep 18 19:52:42.164998 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:52:42.164931    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967\": container with ID starting with 6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967 not found: ID does not exist" containerID="6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"
Sep 18 19:52:42.164998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.164951    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967"} err="failed to get container status \"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967\": rpc error: code = NotFound desc = could not find container \"6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967\": container with ID starting with 6e01df6b87a5f3a7106734469a53c1aac4eeafbc479d422dbdaf29a09649b967 not found: ID does not exist"
Sep 18 19:52:42.964945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.964910    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965215    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965226    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965234    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="storage-initializer"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965240    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="storage-initializer"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965253    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965260    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965266    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965271    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965282    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965287    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965293    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="storage-initializer"
Sep 18 19:52:42.965337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965299    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="storage-initializer"
Sep 18 19:52:42.965763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965374    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kserve-container"
Sep 18 19:52:42.965763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965382    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965388    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" containerName="kube-rbac-proxy"
Sep 18 19:52:42.965763 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.965397    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="840281db-bb3d-4958-a547-b93a1fe8ab0e" containerName="kserve-container"
Sep 18 19:52:42.969020 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.968988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.973377 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.973337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.973506 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.973421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.973565 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.973531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmttl\" (UniqueName: \"kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.973624 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.973598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.973682 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.973667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:42.982013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.981985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 19:52:42.982013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.981999    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:52:42.982188 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.982020    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:52:42.982188 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.982113    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:52:42.982992 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.982975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:52:42.983101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.982989    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 19:52:42.983101 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.983014    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:52:42.997812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:42.997782    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:52:43.074224 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.074458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.074458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmttl\" (UniqueName: \"kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.074458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.074458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.074458 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:52:43.074394    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:52:43.074739 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:52:43.074464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls podName:d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a nodeName:}" failed. No retries permitted until 2026-09-18 19:52:43.574444532 +0000 UTC m=+2295.949870816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 18 19:52:43.074739 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.075003 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.074984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.075048 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.075001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.093158 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.093118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmttl\" (UniqueName: \"kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.577084 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.577044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.579618 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.579598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-574749c459-466bc\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.581500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.581473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:43.791308 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.791274    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc" path="/var/lib/kubelet/pods/c6a0a316-004d-4c64-b2f0-8da6f6f5ebbc/volumes"
Sep 18 19:52:43.874280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:43.874256    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:52:43.876539 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:52:43.876508    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2acdfd3_d53a_4c4d_aa7a_8c5ceba9777a.slice/crio-fbb6e7b62846fb912472456fe1036500d0910610e827671cc7591be1b93b2521 WatchSource:0}: Error finding container fbb6e7b62846fb912472456fe1036500d0910610e827671cc7591be1b93b2521: Status 404 returned error can't find the container with id fbb6e7b62846fb912472456fe1036500d0910610e827671cc7591be1b93b2521
Sep 18 19:52:44.148212 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:44.148119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerStarted","Data":"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"}
Sep 18 19:52:44.148212 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:44.148179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerStarted","Data":"fbb6e7b62846fb912472456fe1036500d0910610e827671cc7591be1b93b2521"}
Sep 18 19:52:48.168781 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:48.168746    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerID="184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b" exitCode=0
Sep 18 19:52:48.169162 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:48.168820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerDied","Data":"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"}
Sep 18 19:52:49.174954 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:49.174916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerStarted","Data":"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"}
Sep 18 19:52:49.175351 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:49.175015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerStarted","Data":"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"}
Sep 18 19:52:49.175351 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:49.175224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:49.220861 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:49.220805    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" podStartSLOduration=3.208351206 podStartE2EDuration="7.220788301s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.012437095s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:49.219937268 +0000 UTC m=+2301.595363561" watchObservedRunningTime="2026-09-18 19:52:49.220788301 +0000 UTC m=+2301.596214594"
Sep 18 19:52:50.179563 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:50.179531    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:52:56.189847 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:52:56.189814    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:53:26.199446 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:26.199338    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:53:32.897895 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:32.897861    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:53:32.898500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:32.898124    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kserve-container" containerID="cri-o://a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5" gracePeriod=30
Sep 18 19:53:32.898500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:32.898141    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kube-rbac-proxy" containerID="cri-o://a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616" gracePeriod=30
Sep 18 19:53:33.362164 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:33.362124    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerID="a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616" exitCode=2
Sep 18 19:53:33.362356 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:33.362194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerDied","Data":"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"}
Sep 18 19:53:36.184794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:36.184737    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 18 19:53:36.190523 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:36.190491    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 18 19:53:38.322544 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.322506    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:53:38.326320 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.326304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.335998 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.335976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 18 19:53:38.336229 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.336212    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 18 19:53:38.359080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.359048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.359080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.359089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.359297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.359118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.359297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.359199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6q6\" (UniqueName: \"kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.359297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.359250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.364112 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.364079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:53:38.459830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.459791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.459830 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.459837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460057 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.459865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.460047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6q6\" (UniqueName: \"kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460168 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.460123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460219 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.460186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.460568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.460683 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.460673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.462403 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.462353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.470695 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.470671    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6q6\" (UniqueName: \"kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6\") pod \"xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.635999 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.635903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:38.933146 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:38.933069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:53:38.936606 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:53:38.936576    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7e2dbc_a617_4668_b9e9_9bdb5b269f7f.slice/crio-7f786795f09f25f5e77a04dd27bd6cac83bad1aba5292d3bca573ad62b343031 WatchSource:0}: Error finding container 7f786795f09f25f5e77a04dd27bd6cac83bad1aba5292d3bca573ad62b343031: Status 404 returned error can't find the container with id 7f786795f09f25f5e77a04dd27bd6cac83bad1aba5292d3bca573ad62b343031
Sep 18 19:53:39.389730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.389695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerStarted","Data":"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"}
Sep 18 19:53:39.390096 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.389767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerStarted","Data":"7f786795f09f25f5e77a04dd27bd6cac83bad1aba5292d3bca573ad62b343031"}
Sep 18 19:53:39.735567 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.735542    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:53:39.770975 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.770947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config\") pod \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") "
Sep 18 19:53:39.771130 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771001    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert\") pod \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") "
Sep 18 19:53:39.771130 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") pod \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") "
Sep 18 19:53:39.771130 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771071    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmttl\" (UniqueName: \"kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl\") pod \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") "
Sep 18 19:53:39.771130 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location\") pod \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\" (UID: \"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a\") "
Sep 18 19:53:39.771441 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771406    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:39.771554 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:39.771554 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.771489    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:39.773201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.773179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:39.773265 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.773217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl" (OuterVolumeSpecName: "kube-api-access-zmttl") pod "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" (UID: "d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a"). InnerVolumeSpecName "kube-api-access-zmttl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:39.871658 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.871614    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:53:39.871658 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.871652    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmttl\" (UniqueName: \"kubernetes.io/projected/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-api-access-zmttl\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:53:39.871658 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.871666    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:53:39.871910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.871679    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:53:39.871910 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:39.871692    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:53:40.395262 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.395223    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerID="a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5" exitCode=0
Sep 18 19:53:40.395774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.395278    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"
Sep 18 19:53:40.395774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.395284    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerDied","Data":"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"}
Sep 18 19:53:40.395774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.395355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc" event={"ID":"d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a","Type":"ContainerDied","Data":"fbb6e7b62846fb912472456fe1036500d0910610e827671cc7591be1b93b2521"}
Sep 18 19:53:40.395774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.395416    2583 scope.go:117] "RemoveContainer" containerID="a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"
Sep 18 19:53:40.406812 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.406774    2583 scope.go:117] "RemoveContainer" containerID="a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"
Sep 18 19:53:40.414452 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.414433    2583 scope.go:117] "RemoveContainer" containerID="184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"
Sep 18 19:53:40.421561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.421546    2583 scope.go:117] "RemoveContainer" containerID="a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"
Sep 18 19:53:40.421787 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:53:40.421768    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616\": container with ID starting with a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616 not found: ID does not exist" containerID="a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"
Sep 18 19:53:40.421865 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.421800    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616"} err="failed to get container status \"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616\": rpc error: code = NotFound desc = could not find container \"a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616\": container with ID starting with a24bf09905e8bf199af4482d00e2895d71256f123243e5aaf8c057d31d966616 not found: ID does not exist"
Sep 18 19:53:40.421865 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.421826    2583 scope.go:117] "RemoveContainer" containerID="a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"
Sep 18 19:53:40.422104 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:53:40.422089    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5\": container with ID starting with a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5 not found: ID does not exist" containerID="a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"
Sep 18 19:53:40.422163 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.422111    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5"} err="failed to get container status \"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5\": rpc error: code = NotFound desc = could not find container \"a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5\": container with ID starting with a4609a90d5c3007b21e69af5d537ea5f4616848a5412e85a1704cd6b04e2beb5 not found: ID does not exist"
Sep 18 19:53:40.422163 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.422132    2583 scope.go:117] "RemoveContainer" containerID="184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"
Sep 18 19:53:40.422391 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:53:40.422355    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b\": container with ID starting with 184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b not found: ID does not exist" containerID="184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"
Sep 18 19:53:40.422505 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.422396    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b"} err="failed to get container status \"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b\": rpc error: code = NotFound desc = could not find container \"184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b\": container with ID starting with 184d12984eea7bd992639deef0fe8c328f364b765d7b1ca969719a25e3fa861b not found: ID does not exist"
Sep 18 19:53:40.425301 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.425279    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:53:40.430100 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:40.430080    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-574749c459-466bc"]
Sep 18 19:53:41.791242 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:41.791205    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" path="/var/lib/kubelet/pods/d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a/volumes"
Sep 18 19:53:43.409876 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:43.409840    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerID="33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7" exitCode=0
Sep 18 19:53:43.410307 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:43.409905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerDied","Data":"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"}
Sep 18 19:53:44.416422 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:44.416385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerStarted","Data":"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"}
Sep 18 19:53:44.416882 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:44.416457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerStarted","Data":"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"}
Sep 18 19:53:44.416882 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:44.416675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:44.455453 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:44.455397    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" podStartSLOduration=2.700030882 podStartE2EDuration="6.45537996s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755349078s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:44.454756331 +0000 UTC m=+2356.830182626" watchObservedRunningTime="2026-09-18 19:53:44.45537996 +0000 UTC m=+2356.830806247"
Sep 18 19:53:45.420842 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:45.420809    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:53:51.430217 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:53:51.430184    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:54:16.099762 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.099727    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100050    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="storage-initializer"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100061    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="storage-initializer"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100068    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kube-rbac-proxy"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100074    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kube-rbac-proxy"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100083    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kserve-container"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100090    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kserve-container"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100141    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kube-rbac-proxy"
Sep 18 19:54:16.100232 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.100150    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2acdfd3-d53a-4c4d-aa7a-8c5ceba9777a" containerName="kserve-container"
Sep 18 19:54:16.103426 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.103408    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.106928 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.106905    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 18 19:54:16.107059 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.106939    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:54:16.118701 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.118671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:54:16.177349 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.177302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmtg\" (UniqueName: \"kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.177590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.177389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.177590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.177430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.177590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.177506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.177590 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.177534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.278920 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.278876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.278920 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.278921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.278961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.278987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.279053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmtg\" (UniqueName: \"kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279190 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:54:16.279158    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 18 19:54:16.279428 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:54:16.279247    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls podName:3eb372a3-12ae-40e0-a871-3f74c2338e6e nodeName:}" failed. No retries permitted until 2026-09-18 19:54:16.779224924 +0000 UTC m=+2389.154651213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls") pod "isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 18 19:54:16.279477 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.279433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279662 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.279641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.279711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.279692    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.289335 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.289306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmtg\" (UniqueName: \"kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.783764 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.783718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:16.786471 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:16.786430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:17.013631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:17.013592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:17.303212 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:17.303181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:54:17.306095 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:54:17.306067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb372a3_12ae_40e0_a871_3f74c2338e6e.slice/crio-a8f2e05776f2adbb23a58f10f8f94a53257833e0d42c4f935b3371658babb040 WatchSource:0}: Error finding container a8f2e05776f2adbb23a58f10f8f94a53257833e0d42c4f935b3371658babb040: Status 404 returned error can't find the container with id a8f2e05776f2adbb23a58f10f8f94a53257833e0d42c4f935b3371658babb040
Sep 18 19:54:17.560279 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:17.560235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerStarted","Data":"29dd0e9c60adc0d6f70871ced5f3cee66bfe2566abd2a2ab5ff5592668f462fb"}
Sep 18 19:54:17.560504 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:17.560319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerStarted","Data":"a8f2e05776f2adbb23a58f10f8f94a53257833e0d42c4f935b3371658babb040"}
Sep 18 19:54:21.434903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:21.434832    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:54:21.579487 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:21.579450    2583 generic.go:356] "Generic (PLEG): container finished" podID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerID="29dd0e9c60adc0d6f70871ced5f3cee66bfe2566abd2a2ab5ff5592668f462fb" exitCode=0
Sep 18 19:54:21.579667 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:21.579523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerDied","Data":"29dd0e9c60adc0d6f70871ced5f3cee66bfe2566abd2a2ab5ff5592668f462fb"}
Sep 18 19:54:21.580927 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:21.580913    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:54:28.092190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.092157    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:54:28.092694 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.092499    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kserve-container" containerID="cri-o://254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7" gracePeriod=30
Sep 18 19:54:28.092907 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.092767    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kube-rbac-proxy" containerID="cri-o://ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75" gracePeriod=30
Sep 18 19:54:28.616494 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.616454    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerID="ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75" exitCode=2
Sep 18 19:54:28.616687 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.616524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerDied","Data":"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"}
Sep 18 19:54:28.618724 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.618702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerStarted","Data":"f8b5d0efd94ef9f9f15993f1ecc5ecf4b29c83c43532eb7f4566213c2884a8fa"}
Sep 18 19:54:28.618837 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.618753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerStarted","Data":"06bd61280d55f2a03a694b98262fa0aeb926d34af13b7c60d03b061e15584a9e"}
Sep 18 19:54:28.618955 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.618933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:28.642248 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:28.642191    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podStartSLOduration=2.722601641 podStartE2EDuration="12.642170656s" totalImagesPullingTime="6.157327902s" totalInitContainerRuntime="3.762241113s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:28.640471959 +0000 UTC m=+2401.015898251" watchObservedRunningTime="2026-09-18 19:54:28.642170656 +0000 UTC m=+2401.017596948"
Sep 18 19:54:29.623614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:29.623576    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:29.624793 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:29.624761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:54:30.627858 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:30.627807    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:54:31.425957 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:31.425915    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 18 19:54:31.431423 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:31.431381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 18 19:54:33.378592 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.378548    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:54:33.382379 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.382348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.394164 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.394144    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:54:33.394694 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.394680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 18 19:54:33.405378 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.405338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:54:33.530894 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.530855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.530894 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.530900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.531106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.530932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.531106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.531008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c794\" (UniqueName: \"kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.531106 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.531045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c794\" (UniqueName: \"kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632529 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632529 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.632978 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.633075 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.632954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.634778 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.634759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.642462 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.642441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c794\" (UniqueName: \"kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794\") pod \"isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.691914 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.691859    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:33.975593 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:33.975569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:54:33.977541 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:54:33.977515    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a61a907_dcdd_4b6e_8c4b_147b50aa2100.slice/crio-bb83ddbaaa8f6492dbfebd5d7a86402ee5525ac0616c9141413b7566c5010cf3 WatchSource:0}: Error finding container bb83ddbaaa8f6492dbfebd5d7a86402ee5525ac0616c9141413b7566c5010cf3: Status 404 returned error can't find the container with id bb83ddbaaa8f6492dbfebd5d7a86402ee5525ac0616c9141413b7566c5010cf3
Sep 18 19:54:34.648334 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:34.648301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerStarted","Data":"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"}
Sep 18 19:54:34.648766 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:34.648373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerStarted","Data":"bb83ddbaaa8f6492dbfebd5d7a86402ee5525ac0616c9141413b7566c5010cf3"}
Sep 18 19:54:35.039806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.039781    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:54:35.147079 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st6q6\" (UniqueName: \"kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6\") pod \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") "
Sep 18 19:54:35.147255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls\") pod \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") "
Sep 18 19:54:35.147255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location\") pod \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") "
Sep 18 19:54:35.147255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147238    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert\") pod \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") "
Sep 18 19:54:35.147430 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config\") pod \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\" (UID: \"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f\") "
Sep 18 19:54:35.147614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147572    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" (UID: "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:35.147689 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147665    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" (UID: "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:35.147742 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.147727    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" (UID: "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:35.149246 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.149222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" (UID: "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:35.149382 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.149343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6" (OuterVolumeSpecName: "kube-api-access-st6q6") pod "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" (UID: "7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f"). InnerVolumeSpecName "kube-api-access-st6q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:35.248303 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.248276    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:54:35.248303 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.248301    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6q6\" (UniqueName: \"kubernetes.io/projected/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kube-api-access-st6q6\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:54:35.248303 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.248312    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:54:35.248575 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.248322    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:54:35.248575 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.248331    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:54:35.632982 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.632882    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:54:35.633545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.633512    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:54:35.654848 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.654809    2583 generic.go:356] "Generic (PLEG): container finished" podID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerID="254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7" exitCode=0
Sep 18 19:54:35.655311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.654876    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"
Sep 18 19:54:35.655311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.654896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerDied","Data":"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"}
Sep 18 19:54:35.655311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.654954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g" event={"ID":"7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f","Type":"ContainerDied","Data":"7f786795f09f25f5e77a04dd27bd6cac83bad1aba5292d3bca573ad62b343031"}
Sep 18 19:54:35.655311 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.654970    2583 scope.go:117] "RemoveContainer" containerID="ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"
Sep 18 19:54:35.664463 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.664435    2583 scope.go:117] "RemoveContainer" containerID="254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"
Sep 18 19:54:35.672619 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.672596    2583 scope.go:117] "RemoveContainer" containerID="33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"
Sep 18 19:54:35.680160 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.680131    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:54:35.680975 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.680960    2583 scope.go:117] "RemoveContainer" containerID="ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"
Sep 18 19:54:35.681295 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:54:35.681275    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75\": container with ID starting with ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75 not found: ID does not exist" containerID="ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"
Sep 18 19:54:35.681383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.681311    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75"} err="failed to get container status \"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75\": rpc error: code = NotFound desc = could not find container \"ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75\": container with ID starting with ca0ebbceca7bcef3db80eee18b8b9da08d38c08b92f6a53723259ed5a00f0c75 not found: ID does not exist"
Sep 18 19:54:35.681383 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.681338    2583 scope.go:117] "RemoveContainer" containerID="254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"
Sep 18 19:54:35.681641 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:54:35.681625    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7\": container with ID starting with 254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7 not found: ID does not exist" containerID="254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"
Sep 18 19:54:35.681690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.681646    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7"} err="failed to get container status \"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7\": rpc error: code = NotFound desc = could not find container \"254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7\": container with ID starting with 254698c2a816b0fb22a5dd7e49d2fb57fe3fc8696c7c7efd485fc8a4948d7dd7 not found: ID does not exist"
Sep 18 19:54:35.681690 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.681661    2583 scope.go:117] "RemoveContainer" containerID="33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"
Sep 18 19:54:35.681877 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:54:35.681859    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7\": container with ID starting with 33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7 not found: ID does not exist" containerID="33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"
Sep 18 19:54:35.681923 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.681879    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7"} err="failed to get container status \"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7\": rpc error: code = NotFound desc = could not find container \"33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7\": container with ID starting with 33dd99284ba4c3562144d9ff74f73ba3929811836c40d2cb0a69e26cacffdad7 not found: ID does not exist"
Sep 18 19:54:35.685500 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.685475    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-6d58fc45fb-d5c5g"]
Sep 18 19:54:35.791860 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:35.791824    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" path="/var/lib/kubelet/pods/7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f/volumes"
Sep 18 19:54:38.669988 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:38.669952    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerID="bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6" exitCode=0
Sep 18 19:54:38.670549 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:38.670030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerDied","Data":"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"}
Sep 18 19:54:39.676549 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.676515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerStarted","Data":"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"}
Sep 18 19:54:39.676937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.676579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerStarted","Data":"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"}
Sep 18 19:54:39.676937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.676842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:39.677072 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.677001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:39.678234 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.678207    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:54:39.710751 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:39.710696    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podStartSLOduration=2.8019222299999997 podStartE2EDuration="6.710675982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908753752s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:39.710333852 +0000 UTC m=+2412.085760145" watchObservedRunningTime="2026-09-18 19:54:39.710675982 +0000 UTC m=+2412.086102276"
Sep 18 19:54:40.681444 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:40.681348    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:54:45.633841 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:45.633801    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:54:45.686230 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:45.686201    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:54:45.686744 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:45.686714    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:54:55.633423 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:55.633382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:54:55.687468 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:54:55.687430    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:05.633667 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:05.633623    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:05.687512 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:05.687468    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:15.634043 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:15.633994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:15.687610 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:15.687569    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:25.634280 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:25.634234    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:25.687677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:25.687641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:29.219137 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:29.219109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:55:29.221578 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:29.221554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 19:55:35.634175 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:35.634130    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:35.687435 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:35.687391    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:45.634223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:45.634177    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:45.686936 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:45.686897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 18 19:55:48.786711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:48.786668    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:55:53.791423 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:53.791395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:55:58.786915 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:55:58.786868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:56:04.027282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:04.027241    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:56:04.027838 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:04.027508    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container" containerID="cri-o://24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da" gracePeriod=30
Sep 18 19:56:04.027838 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:04.027539    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy" containerID="cri-o://aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae" gracePeriod=30
Sep 18 19:56:05.060946 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:05.060912    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerID="aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae" exitCode=2
Sep 18 19:56:05.061349 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:05.060992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerDied","Data":"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"}
Sep 18 19:56:05.682030 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:05.681982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 19:56:08.787415 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:08.787347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:56:09.170464 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170377    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:56:09.170771 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170757    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kube-rbac-proxy"
Sep 18 19:56:09.170816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170774    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kube-rbac-proxy"
Sep 18 19:56:09.170816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170784    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kserve-container"
Sep 18 19:56:09.170816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170790    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kserve-container"
Sep 18 19:56:09.170816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170802    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="storage-initializer"
Sep 18 19:56:09.170816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170808    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="storage-initializer"
Sep 18 19:56:09.170973 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170867    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kube-rbac-proxy"
Sep 18 19:56:09.170973 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.170875    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e7e2dbc-a617-4668-b9e9-9bdb5b269f7f" containerName="kserve-container"
Sep 18 19:56:09.174219 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.174200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.177110 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.177087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 18 19:56:09.177110 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.177104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:56:09.188953 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.188919    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:56:09.256733 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.256681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.256941 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.256754    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.256941 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.256783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqnm\" (UniqueName: \"kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.256941 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.256816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.256941 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.256840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358019 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.357977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358019 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqnm\" (UniqueName: \"kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358453 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:56:09.358283    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 18 19:56:09.358453 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:56:09.358411    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls podName:d6ff96a2-cfd5-45b6-bcaa-0e5664264f53 nodeName:}" failed. No retries permitted until 2026-09-18 19:56:09.858354223 +0000 UTC m=+2502.233780495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 18 19:56:09.358600 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358815 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.358870 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.358801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.368075 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.368041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqnm\" (UniqueName: \"kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.862795 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.862731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:09.865203 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:09.865181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:10.084173 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:10.084137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:10.369162 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:10.369125    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:56:10.370737 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:56:10.370710    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ff96a2_cfd5_45b6_bcaa_0e5664264f53.slice/crio-27a09e3eb9105d31849db6e0102a115d1e944236e66688ee949aa6452153ab8a WatchSource:0}: Error finding container 27a09e3eb9105d31849db6e0102a115d1e944236e66688ee949aa6452153ab8a: Status 404 returned error can't find the container with id 27a09e3eb9105d31849db6e0102a115d1e944236e66688ee949aa6452153ab8a
Sep 18 19:56:10.682718 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:10.682615    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 18 19:56:11.089545 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.089505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerStarted","Data":"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"}
Sep 18 19:56:11.089955 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.089567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerStarted","Data":"27a09e3eb9105d31849db6e0102a115d1e944236e66688ee949aa6452153ab8a"}
Sep 18 19:56:11.880090 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.880064    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:56:11.985302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985209    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert\") pod \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") "
Sep 18 19:56:11.985302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985248    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config\") pod \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") "
Sep 18 19:56:11.985302 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985271    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location\") pod \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") "
Sep 18 19:56:11.985641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls\") pod \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") "
Sep 18 19:56:11.985641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c794\" (UniqueName: \"kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794\") pod \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\" (UID: \"9a61a907-dcdd-4b6e-8c4b-147b50aa2100\") "
Sep 18 19:56:11.985641 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985592    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a61a907-dcdd-4b6e-8c4b-147b50aa2100" (UID: "9a61a907-dcdd-4b6e-8c4b-147b50aa2100"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:11.985774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a61a907-dcdd-4b6e-8c4b-147b50aa2100" (UID: "9a61a907-dcdd-4b6e-8c4b-147b50aa2100"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:11.985774 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.985689    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a61a907-dcdd-4b6e-8c4b-147b50aa2100" (UID: "9a61a907-dcdd-4b6e-8c4b-147b50aa2100"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:11.987543 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.987519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a61a907-dcdd-4b6e-8c4b-147b50aa2100" (UID: "9a61a907-dcdd-4b6e-8c4b-147b50aa2100"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:11.987543 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:11.987514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794" (OuterVolumeSpecName: "kube-api-access-6c794") pod "9a61a907-dcdd-4b6e-8c4b-147b50aa2100" (UID: "9a61a907-dcdd-4b6e-8c4b-147b50aa2100"). InnerVolumeSpecName "kube-api-access-6c794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:12.087015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.086973    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.087015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.087010    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c794\" (UniqueName: \"kubernetes.io/projected/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-api-access-6c794\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.087015 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.087020    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.087272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.087029    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.087272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.087039    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a61a907-dcdd-4b6e-8c4b-147b50aa2100-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:12.095970 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.095931    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerID="24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da" exitCode=0
Sep 18 19:56:12.096491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.095993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"
Sep 18 19:56:12.096491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.096020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerDied","Data":"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"}
Sep 18 19:56:12.096491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.096099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps" event={"ID":"9a61a907-dcdd-4b6e-8c4b-147b50aa2100","Type":"ContainerDied","Data":"bb83ddbaaa8f6492dbfebd5d7a86402ee5525ac0616c9141413b7566c5010cf3"}
Sep 18 19:56:12.096491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.096143    2583 scope.go:117] "RemoveContainer" containerID="aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"
Sep 18 19:56:12.106448 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.106428    2583 scope.go:117] "RemoveContainer" containerID="24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"
Sep 18 19:56:12.115747 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.115722    2583 scope.go:117] "RemoveContainer" containerID="bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"
Sep 18 19:56:12.119903 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.119875    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:56:12.122038 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.122013    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6fdbb69fc8-xqvps"]
Sep 18 19:56:12.124784 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.124766    2583 scope.go:117] "RemoveContainer" containerID="aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"
Sep 18 19:56:12.125149 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:56:12.125130    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae\": container with ID starting with aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae not found: ID does not exist" containerID="aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"
Sep 18 19:56:12.125200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.125159    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae"} err="failed to get container status \"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae\": rpc error: code = NotFound desc = could not find container \"aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae\": container with ID starting with aaaa9fc0f3855b8bcb72769517a1c6ec52de517e11cda30fc545aa88c586aeae not found: ID does not exist"
Sep 18 19:56:12.125200 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.125180    2583 scope.go:117] "RemoveContainer" containerID="24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"
Sep 18 19:56:12.125487 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:56:12.125468    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da\": container with ID starting with 24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da not found: ID does not exist" containerID="24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"
Sep 18 19:56:12.125539 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.125491    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da"} err="failed to get container status \"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da\": rpc error: code = NotFound desc = could not find container \"24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da\": container with ID starting with 24dac0eb8add8ec0f9bc5b6aa60e342a7f729cffe66fdcd376ee664d6f3e15da not found: ID does not exist"
Sep 18 19:56:12.125539 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.125507    2583 scope.go:117] "RemoveContainer" containerID="bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"
Sep 18 19:56:12.125752 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:56:12.125734    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6\": container with ID starting with bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6 not found: ID does not exist" containerID="bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"
Sep 18 19:56:12.125816 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:12.125774    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6"} err="failed to get container status \"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6\": rpc error: code = NotFound desc = could not find container \"bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6\": container with ID starting with bc781105f4821ebb6c65908f13cb8b995c698d33ad7d9a8d16e86559988935a6 not found: ID does not exist"
Sep 18 19:56:13.792155 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:13.792118    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" path="/var/lib/kubelet/pods/9a61a907-dcdd-4b6e-8c4b-147b50aa2100/volumes"
Sep 18 19:56:15.111057 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:15.111025    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerID="939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af" exitCode=0
Sep 18 19:56:15.111477 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:15.111087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerDied","Data":"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"}
Sep 18 19:56:16.118496 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:16.118458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerStarted","Data":"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"}
Sep 18 19:56:16.118990 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:16.118537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerStarted","Data":"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"}
Sep 18 19:56:16.118990 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:16.118755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:16.148053 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:16.147992    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podStartSLOduration=3.386587213 podStartE2EDuration="7.147973655s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761386442s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:16.144757641 +0000 UTC m=+2508.520183934" watchObservedRunningTime="2026-09-18 19:56:16.147973655 +0000 UTC m=+2508.523399947"
Sep 18 19:56:17.123921 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:17.123892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:17.689760 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:17.689723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:56:17.689972 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:17.689941    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" containerID="cri-o://06bd61280d55f2a03a694b98262fa0aeb926d34af13b7c60d03b061e15584a9e" gracePeriod=30
Sep 18 19:56:17.690128 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:17.689992    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kube-rbac-proxy" containerID="cri-o://f8b5d0efd94ef9f9f15993f1ecc5ecf4b29c83c43532eb7f4566213c2884a8fa" gracePeriod=30
Sep 18 19:56:18.130579 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:18.130548    2583 generic.go:356] "Generic (PLEG): container finished" podID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerID="f8b5d0efd94ef9f9f15993f1ecc5ecf4b29c83c43532eb7f4566213c2884a8fa" exitCode=2
Sep 18 19:56:18.131070 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:18.130625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerDied","Data":"f8b5d0efd94ef9f9f15993f1ecc5ecf4b29c83c43532eb7f4566213c2884a8fa"}
Sep 18 19:56:18.787090 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:18.787043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 18 19:56:20.628621 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:20.628576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 18 19:56:22.743262 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743218    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:56:22.743790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743730    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy"
Sep 18 19:56:22.743790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743751    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy"
Sep 18 19:56:22.743790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743774    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container"
Sep 18 19:56:22.743790 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743781    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container"
Sep 18 19:56:22.744011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="storage-initializer"
Sep 18 19:56:22.744011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743814    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="storage-initializer"
Sep 18 19:56:22.744011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743910    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kserve-container"
Sep 18 19:56:22.744011 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.743922    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a61a907-dcdd-4b6e-8c4b-147b50aa2100" containerName="kube-rbac-proxy"
Sep 18 19:56:22.747424 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.747401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.750208 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.750186    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 18 19:56:22.750601 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.750579    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:56:22.761174 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.761146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:56:22.887928 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.887886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.888132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.887936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkl8m\" (UniqueName: \"kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.888132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.887997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.888132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.888024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.888132 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.888111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.988789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.988746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.988789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.988797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkl8m\" (UniqueName: \"kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989077 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.988983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989077 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.989046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.989113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.989327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.989563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.989727 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.989709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:22.991765 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:22.991744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:23.001911 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.001837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkl8m\" (UniqueName: \"kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m\") pod \"isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:23.059611 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.059565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:23.137013 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.136989    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:56:23.154447 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.154414    2583 generic.go:356] "Generic (PLEG): container finished" podID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerID="06bd61280d55f2a03a694b98262fa0aeb926d34af13b7c60d03b061e15584a9e" exitCode=0
Sep 18 19:56:23.154607 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.154465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerDied","Data":"06bd61280d55f2a03a694b98262fa0aeb926d34af13b7c60d03b061e15584a9e"}
Sep 18 19:56:23.348562 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.348535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:56:23.350459 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:56:23.350434    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d8baef_e1dc_46a1_a538_3845da0d2bc5.slice/crio-ea8eafe7f11ebf7d61506bb0956dd446e96bf6a54579a50a42ecccbeefbf572f WatchSource:0}: Error finding container ea8eafe7f11ebf7d61506bb0956dd446e96bf6a54579a50a42ecccbeefbf572f: Status 404 returned error can't find the container with id ea8eafe7f11ebf7d61506bb0956dd446e96bf6a54579a50a42ecccbeefbf572f
Sep 18 19:56:23.525423 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.525392    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:56:23.695583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695494    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert\") pod \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") "
Sep 18 19:56:23.695583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config\") pod \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") "
Sep 18 19:56:23.695809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695601    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") pod \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") "
Sep 18 19:56:23.695809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqmtg\" (UniqueName: \"kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg\") pod \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") "
Sep 18 19:56:23.695809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location\") pod \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\" (UID: \"3eb372a3-12ae-40e0-a871-3f74c2338e6e\") "
Sep 18 19:56:23.695971 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695915    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3eb372a3-12ae-40e0-a871-3f74c2338e6e" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:23.695971 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.695930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3eb372a3-12ae-40e0-a871-3f74c2338e6e" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:23.696347 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.696318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3eb372a3-12ae-40e0-a871-3f74c2338e6e" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:23.698282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.698249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3eb372a3-12ae-40e0-a871-3f74c2338e6e" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:23.698282 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.698259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg" (OuterVolumeSpecName: "kube-api-access-xqmtg") pod "3eb372a3-12ae-40e0-a871-3f74c2338e6e" (UID: "3eb372a3-12ae-40e0-a871-3f74c2338e6e"). InnerVolumeSpecName "kube-api-access-xqmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:23.796201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.796169    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:23.796201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.796194    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:23.796201 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.796204    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3eb372a3-12ae-40e0-a871-3f74c2338e6e-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:23.796640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.796213    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmtg\" (UniqueName: \"kubernetes.io/projected/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kube-api-access-xqmtg\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:23.796640 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:23.796222    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3eb372a3-12ae-40e0-a871-3f74c2338e6e-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:56:24.161626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.161592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"
Sep 18 19:56:24.161626 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.161608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp" event={"ID":"3eb372a3-12ae-40e0-a871-3f74c2338e6e","Type":"ContainerDied","Data":"a8f2e05776f2adbb23a58f10f8f94a53257833e0d42c4f935b3371658babb040"}
Sep 18 19:56:24.161954 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.161724    2583 scope.go:117] "RemoveContainer" containerID="f8b5d0efd94ef9f9f15993f1ecc5ecf4b29c83c43532eb7f4566213c2884a8fa"
Sep 18 19:56:24.163198 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.163175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerStarted","Data":"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"}
Sep 18 19:56:24.163292 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.163229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerStarted","Data":"ea8eafe7f11ebf7d61506bb0956dd446e96bf6a54579a50a42ecccbeefbf572f"}
Sep 18 19:56:24.170634 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.170607    2583 scope.go:117] "RemoveContainer" containerID="06bd61280d55f2a03a694b98262fa0aeb926d34af13b7c60d03b061e15584a9e"
Sep 18 19:56:24.178268 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.178243    2583 scope.go:117] "RemoveContainer" containerID="29dd0e9c60adc0d6f70871ced5f3cee66bfe2566abd2a2ab5ff5592668f462fb"
Sep 18 19:56:24.215880 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.215834    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:56:24.219086 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:24.219053    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-7b5c95bfbc-2pcsp"]
Sep 18 19:56:25.791876 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:25.791845    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" path="/var/lib/kubelet/pods/3eb372a3-12ae-40e0-a871-3f74c2338e6e/volumes"
Sep 18 19:56:28.183278 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:28.183244    2583 generic.go:356] "Generic (PLEG): container finished" podID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerID="7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500" exitCode=0
Sep 18 19:56:28.183693 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:28.183317    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerDied","Data":"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"}
Sep 18 19:56:29.190330 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:29.190294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerStarted","Data":"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"}
Sep 18 19:56:29.190770 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:29.190405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerStarted","Data":"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"}
Sep 18 19:56:29.190770 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:29.190705    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:29.235377 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:29.235313    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" podStartSLOduration=3.126674799 podStartE2EDuration="7.235297095s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.108622296s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:56:29.232596678 +0000 UTC m=+2521.608022973" watchObservedRunningTime="2026-09-18 19:56:29.235297095 +0000 UTC m=+2521.610723388"
Sep 18 19:56:30.196933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:30.196894    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:36.207531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:36.207498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:56:53.200466 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:56:53.200418    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 18 19:57:03.140155 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:03.140122    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:57:06.211217 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:06.211183    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:57:09.186561 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:09.186524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:57:09.187047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:09.186754    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container" containerID="cri-o://9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911" gracePeriod=30
Sep 18 19:57:09.187047 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:09.186810    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kube-rbac-proxy" containerID="cri-o://9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa" gracePeriod=30
Sep 18 19:57:09.374580 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:09.374541    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerID="9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa" exitCode=2
Sep 18 19:57:09.374753 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:09.374618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerDied","Data":"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"}
Sep 18 19:57:12.810679 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:12.810643    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:57:12.811165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:12.810993    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kube-rbac-proxy" containerID="cri-o://0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e" gracePeriod=30
Sep 18 19:57:12.811165 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:12.810993    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kserve-container" containerID="cri-o://c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639" gracePeriod=30
Sep 18 19:57:13.131726 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:13.131621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 18 19:57:13.138213 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:13.138176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 18 19:57:13.401470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:13.401375    2583 generic.go:356] "Generic (PLEG): container finished" podID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerID="0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e" exitCode=2
Sep 18 19:57:13.401470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:13.401399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerDied","Data":"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"}
Sep 18 19:57:13.956692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:13.956661    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:57:14.033190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033152    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config\") pod \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") "
Sep 18 19:57:14.033190 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033192    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert\") pod \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") "
Sep 18 19:57:14.033470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls\") pod \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") "
Sep 18 19:57:14.033470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkl8m\" (UniqueName: \"kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m\") pod \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") "
Sep 18 19:57:14.033470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location\") pod \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\" (UID: \"52d8baef-e1dc-46a1-a538-3845da0d2bc5\") "
Sep 18 19:57:14.033743 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52d8baef-e1dc-46a1-a538-3845da0d2bc5" (UID: "52d8baef-e1dc-46a1-a538-3845da0d2bc5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:14.033806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52d8baef-e1dc-46a1-a538-3845da0d2bc5" (UID: "52d8baef-e1dc-46a1-a538-3845da0d2bc5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:14.033806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.033766    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52d8baef-e1dc-46a1-a538-3845da0d2bc5" (UID: "52d8baef-e1dc-46a1-a538-3845da0d2bc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:14.035591 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.035568    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52d8baef-e1dc-46a1-a538-3845da0d2bc5" (UID: "52d8baef-e1dc-46a1-a538-3845da0d2bc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:14.035657 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.035569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m" (OuterVolumeSpecName: "kube-api-access-wkl8m") pod "52d8baef-e1dc-46a1-a538-3845da0d2bc5" (UID: "52d8baef-e1dc-46a1-a538-3845da0d2bc5"). InnerVolumeSpecName "kube-api-access-wkl8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:14.134614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.134511    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:14.134614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.134558    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52d8baef-e1dc-46a1-a538-3845da0d2bc5-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:14.134614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.134571    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52d8baef-e1dc-46a1-a538-3845da0d2bc5-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:14.134614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.134580    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkl8m\" (UniqueName: \"kubernetes.io/projected/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kube-api-access-wkl8m\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:14.134614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.134589    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52d8baef-e1dc-46a1-a538-3845da0d2bc5-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:14.407607 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.407517    2583 generic.go:356] "Generic (PLEG): container finished" podID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerID="c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639" exitCode=0
Sep 18 19:57:14.407607 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.407580    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"
Sep 18 19:57:14.407607 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.407590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerDied","Data":"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"}
Sep 18 19:57:14.407945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.407664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx" event={"ID":"52d8baef-e1dc-46a1-a538-3845da0d2bc5","Type":"ContainerDied","Data":"ea8eafe7f11ebf7d61506bb0956dd446e96bf6a54579a50a42ecccbeefbf572f"}
Sep 18 19:57:14.407945 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.407693    2583 scope.go:117] "RemoveContainer" containerID="0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"
Sep 18 19:57:14.416669 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.416650    2583 scope.go:117] "RemoveContainer" containerID="c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"
Sep 18 19:57:14.424814 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.424794    2583 scope.go:117] "RemoveContainer" containerID="7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"
Sep 18 19:57:14.432819 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.432795    2583 scope.go:117] "RemoveContainer" containerID="0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"
Sep 18 19:57:14.433184 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:14.433162    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e\": container with ID starting with 0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e not found: ID does not exist" containerID="0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"
Sep 18 19:57:14.433251 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.433193    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e"} err="failed to get container status \"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e\": rpc error: code = NotFound desc = could not find container \"0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e\": container with ID starting with 0a12da0fb4d68b02901e23ee442b7ba0c256f2a1a45e62990d956544fd8f412e not found: ID does not exist"
Sep 18 19:57:14.433251 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.433212    2583 scope.go:117] "RemoveContainer" containerID="c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"
Sep 18 19:57:14.433544 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:14.433519    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639\": container with ID starting with c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639 not found: ID does not exist" containerID="c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"
Sep 18 19:57:14.433614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.433552    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639"} err="failed to get container status \"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639\": rpc error: code = NotFound desc = could not find container \"c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639\": container with ID starting with c6839d649dcdcf52422e6954e7248ebc284188446bb8224e33f841c134c01639 not found: ID does not exist"
Sep 18 19:57:14.433614 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.433570    2583 scope.go:117] "RemoveContainer" containerID="7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"
Sep 18 19:57:14.433816 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:14.433794    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500\": container with ID starting with 7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500 not found: ID does not exist" containerID="7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"
Sep 18 19:57:14.433872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.433825    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500"} err="failed to get container status \"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500\": rpc error: code = NotFound desc = could not find container \"7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500\": container with ID starting with 7f7be438f23b554b2d082aece46cecff747d27a215d270b15ab26bee270a8500 not found: ID does not exist"
Sep 18 19:57:14.443791 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.443761    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:57:14.444090 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444079    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444092    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444103    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="storage-initializer"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444109    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="storage-initializer"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444118    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444124    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444132    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="storage-initializer"
Sep 18 19:57:14.444136 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444137    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="storage-initializer"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444146    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444151    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444163    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kserve-container"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444168    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kserve-container"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444222    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444231    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kube-rbac-proxy"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" containerName="kserve-container"
Sep 18 19:57:14.444413 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.444244    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb372a3-12ae-40e0-a871-3f74c2338e6e" containerName="kserve-container"
Sep 18 19:57:14.449062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.449037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.451245 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.451218    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:57:14.452286 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.452266    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 18 19:57:14.452403 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.452282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 18 19:57:14.462470 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.462445    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-58f9f86dd8-trxtx"]
Sep 18 19:57:14.466585 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.466560    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:57:14.538382 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.538334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.538569 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.538463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.538569 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.538527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.538650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.538573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52hc\" (UniqueName: \"kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.538650 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.538603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.639583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.639546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52hc\" (UniqueName: \"kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.639583 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.639588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.639799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.639633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.639799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.639666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.639799 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.639699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.640086 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.640067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.640443 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.640415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.640551 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.640476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.642284 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.642264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.650531 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.650501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52hc\" (UniqueName: \"kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc\") pod \"isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:14.759289 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:14.759251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:15.057955 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:15.057793    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:57:15.060467 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:57:15.060437    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3740fbd_7f4c_499c_a2a7_5742957aa3f3.slice/crio-8a501d4ff7dd80b9b44885ddbd5e2183c48355e8f0b994d14897864f59f5022d WatchSource:0}: Error finding container 8a501d4ff7dd80b9b44885ddbd5e2183c48355e8f0b994d14897864f59f5022d: Status 404 returned error can't find the container with id 8a501d4ff7dd80b9b44885ddbd5e2183c48355e8f0b994d14897864f59f5022d
Sep 18 19:57:15.414310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:15.414196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerStarted","Data":"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"}
Sep 18 19:57:15.414310 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:15.414278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerStarted","Data":"8a501d4ff7dd80b9b44885ddbd5e2183c48355e8f0b994d14897864f59f5022d"}
Sep 18 19:57:15.792211 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:15.792176    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d8baef-e1dc-46a1-a538-3845da0d2bc5" path="/var/lib/kubelet/pods/52d8baef-e1dc-46a1-a538-3845da0d2bc5/volumes"
Sep 18 19:57:17.229323 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.229293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:57:17.363080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.362981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert\") pod \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") "
Sep 18 19:57:17.363080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363028    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config\") pod \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") "
Sep 18 19:57:17.363080 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqqnm\" (UniqueName: \"kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm\") pod \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") "
Sep 18 19:57:17.363411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location\") pod \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") "
Sep 18 19:57:17.363411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") pod \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\" (UID: \"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53\") "
Sep 18 19:57:17.363530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:17.363530 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:17.363609 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.363524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:17.365252 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.365233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:17.365337 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.365318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm" (OuterVolumeSpecName: "kube-api-access-gqqnm") pod "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" (UID: "d6ff96a2-cfd5-45b6-bcaa-0e5664264f53"). InnerVolumeSpecName "kube-api-access-gqqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:17.426743 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.426701    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerID="9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911" exitCode=0
Sep 18 19:57:17.426919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.426776    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"
Sep 18 19:57:17.426919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.426795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerDied","Data":"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"}
Sep 18 19:57:17.426919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.426858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5" event={"ID":"d6ff96a2-cfd5-45b6-bcaa-0e5664264f53","Type":"ContainerDied","Data":"27a09e3eb9105d31849db6e0102a115d1e944236e66688ee949aa6452153ab8a"}
Sep 18 19:57:17.426919 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.426888    2583 scope.go:117] "RemoveContainer" containerID="9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"
Sep 18 19:57:17.439481 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.436640    2583 scope.go:117] "RemoveContainer" containerID="9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"
Sep 18 19:57:17.447700 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.447674    2583 scope.go:117] "RemoveContainer" containerID="939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"
Sep 18 19:57:17.455208 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.455189    2583 scope.go:117] "RemoveContainer" containerID="9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"
Sep 18 19:57:17.455517 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:17.455498    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa\": container with ID starting with 9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa not found: ID does not exist" containerID="9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"
Sep 18 19:57:17.455570 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.455527    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa"} err="failed to get container status \"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa\": rpc error: code = NotFound desc = could not find container \"9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa\": container with ID starting with 9f01842a57f13cb10c5e2fd2933d9b92abd5c4786d140bf27002044a6c1d78aa not found: ID does not exist"
Sep 18 19:57:17.455570 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.455545    2583 scope.go:117] "RemoveContainer" containerID="9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"
Sep 18 19:57:17.455836 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:17.455809    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911\": container with ID starting with 9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911 not found: ID does not exist" containerID="9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"
Sep 18 19:57:17.455879 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.455845    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911"} err="failed to get container status \"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911\": rpc error: code = NotFound desc = could not find container \"9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911\": container with ID starting with 9022fb57121cf12decf98ac79c4aacc6361402c38218bbd2b347ea67935bf911 not found: ID does not exist"
Sep 18 19:57:17.455879 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.455863    2583 scope.go:117] "RemoveContainer" containerID="939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"
Sep 18 19:57:17.456121 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:17.456104    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af\": container with ID starting with 939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af not found: ID does not exist" containerID="939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"
Sep 18 19:57:17.456166 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.456128    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af"} err="failed to get container status \"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af\": rpc error: code = NotFound desc = could not find container \"939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af\": container with ID starting with 939cea110741b2773176711977f02f8dd68b62a354c64f77a741014ff430c6af not found: ID does not exist"
Sep 18 19:57:17.463796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.463774    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:17.463796 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.463796    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:17.463933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.463806    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:17.463933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.463817    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqnm\" (UniqueName: \"kubernetes.io/projected/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kube-api-access-gqqnm\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:17.463933 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.463827    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:17.464420 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.464399    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:57:17.467468 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.467447    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6b7777c6-2qpv5"]
Sep 18 19:57:17.792744 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.792714    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" path="/var/lib/kubelet/pods/d6ff96a2-cfd5-45b6-bcaa-0e5664264f53/volumes"
Sep 18 19:57:17.968909 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.968872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:17.969197 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969185    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container"
Sep 18 19:57:17.969197 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969199    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container"
Sep 18 19:57:17.969297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969214    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="storage-initializer"
Sep 18 19:57:17.969297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969220    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="storage-initializer"
Sep 18 19:57:17.969297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969230    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kube-rbac-proxy"
Sep 18 19:57:17.969297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969235    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kube-rbac-proxy"
Sep 18 19:57:17.969297 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969292    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kube-rbac-proxy"
Sep 18 19:57:17.969492 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.969305    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ff96a2-cfd5-45b6-bcaa-0e5664264f53" containerName="kserve-container"
Sep 18 19:57:17.974178 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.974161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:17.977827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.977804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 18 19:57:17.977827 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.977825    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 19:57:17.986029 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:17.986007    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:18.068930 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.068837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.068930 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.068882    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cx55\" (UniqueName: \"kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.069123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.068957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.069123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.068998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.069123 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.069027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.169677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.169637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.169677 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.169682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cx55\" (UniqueName: \"kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.169939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.169719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.169939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.169743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.169939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.169777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.170385 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.170326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.170534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.170406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.170534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.170457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.172258 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.172234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.179519 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.179497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cx55\" (UniqueName: \"kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55\") pod \"isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.284020 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.283977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:18.559973 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:18.559941    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:18.561298 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:57:18.561267    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5010ea9c_5364_4a94_b497_70dffd464e18.slice/crio-f7f7a4006c44a6e94b79071fa73e4269a25f7d0e272d6c0d0d3180bce613ee4c WatchSource:0}: Error finding container f7f7a4006c44a6e94b79071fa73e4269a25f7d0e272d6c0d0d3180bce613ee4c: Status 404 returned error can't find the container with id f7f7a4006c44a6e94b79071fa73e4269a25f7d0e272d6c0d0d3180bce613ee4c
Sep 18 19:57:19.440169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:19.440070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerStarted","Data":"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"}
Sep 18 19:57:19.440169 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:19.440133    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerStarted","Data":"f7f7a4006c44a6e94b79071fa73e4269a25f7d0e272d6c0d0d3180bce613ee4c"}
Sep 18 19:57:19.441678 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:19.441651    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerID="6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4" exitCode=0
Sep 18 19:57:19.441784 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:19.441725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerDied","Data":"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"}
Sep 18 19:57:20.449223 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:20.449184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerStarted","Data":"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"}
Sep 18 19:57:20.449664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:20.449250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerStarted","Data":"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"}
Sep 18 19:57:20.477051 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:20.476992    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podStartSLOduration=2.534147063 podStartE2EDuration="6.476975017s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942827954s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:20.476340685 +0000 UTC m=+2572.851766977" watchObservedRunningTime="2026-09-18 19:57:20.476975017 +0000 UTC m=+2572.852401309"
Sep 18 19:57:23.468968 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:23.468923    2583 generic.go:356] "Generic (PLEG): container finished" podID="5010ea9c-5364-4a94-b497-70dffd464e18" containerID="c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352" exitCode=0
Sep 18 19:57:23.469491 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:23.469001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerDied","Data":"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"}
Sep 18 19:57:24.475872 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:24.475836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerStarted","Data":"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"}
Sep 18 19:57:24.476341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:24.475909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerStarted","Data":"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"}
Sep 18 19:57:24.476341 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:24.476124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:24.507467 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:24.507395    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podStartSLOduration=3.641272218 podStartE2EDuration="7.507354431s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.866082213s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:24.504517583 +0000 UTC m=+2576.879943875" watchObservedRunningTime="2026-09-18 19:57:24.507354431 +0000 UTC m=+2576.882780725"
Sep 18 19:57:25.450539 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.450498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:25.450979 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.450942    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:25.452156 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.452125    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:25.455857 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.455835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:57:25.481668 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.481622    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:25.482170 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.481744    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:25.482876 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:25.482848    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 19:57:26.487272 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:26.487225    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:26.487697 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:26.487227    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 19:57:31.491730 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:31.491696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:31.492255 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:31.492210    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 18 19:57:36.487625 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:36.487574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:41.492571 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:41.492490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:46.488141 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:46.488084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:48.097884 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:48.097842    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:48.098328 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:48.098152    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container" containerID="cri-o://3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5" gracePeriod=30
Sep 18 19:57:48.098328 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:48.098208    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kube-rbac-proxy" containerID="cri-o://297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4" gracePeriod=30
Sep 18 19:57:48.596527 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:48.596492    2583 generic.go:356] "Generic (PLEG): container finished" podID="5010ea9c-5364-4a94-b497-70dffd464e18" containerID="297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4" exitCode=2
Sep 18 19:57:48.596527 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:48.596518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerDied","Data":"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"}
Sep 18 19:57:49.335158 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.335135    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:49.443348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location\") pod \"5010ea9c-5364-4a94-b497-70dffd464e18\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") "
Sep 18 19:57:49.443348 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert\") pod \"5010ea9c-5364-4a94-b497-70dffd464e18\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") "
Sep 18 19:57:49.443649 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443449    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls\") pod \"5010ea9c-5364-4a94-b497-70dffd464e18\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") "
Sep 18 19:57:49.443649 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config\") pod \"5010ea9c-5364-4a94-b497-70dffd464e18\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") "
Sep 18 19:57:49.443649 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443547    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cx55\" (UniqueName: \"kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55\") pod \"5010ea9c-5364-4a94-b497-70dffd464e18\" (UID: \"5010ea9c-5364-4a94-b497-70dffd464e18\") "
Sep 18 19:57:49.443794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443717    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5010ea9c-5364-4a94-b497-70dffd464e18" (UID: "5010ea9c-5364-4a94-b497-70dffd464e18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:49.443794 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5010ea9c-5364-4a94-b497-70dffd464e18" (UID: "5010ea9c-5364-4a94-b497-70dffd464e18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:49.443877 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.443825    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5010ea9c-5364-4a94-b497-70dffd464e18" (UID: "5010ea9c-5364-4a94-b497-70dffd464e18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:49.445624 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.445602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55" (OuterVolumeSpecName: "kube-api-access-5cx55") pod "5010ea9c-5364-4a94-b497-70dffd464e18" (UID: "5010ea9c-5364-4a94-b497-70dffd464e18"). InnerVolumeSpecName "kube-api-access-5cx55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:49.445684 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.445670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5010ea9c-5364-4a94-b497-70dffd464e18" (UID: "5010ea9c-5364-4a94-b497-70dffd464e18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:49.544867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.544830    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cx55\" (UniqueName: \"kubernetes.io/projected/5010ea9c-5364-4a94-b497-70dffd464e18-kube-api-access-5cx55\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:49.544867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.544860    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5010ea9c-5364-4a94-b497-70dffd464e18-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:49.544867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.544869    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:49.544867 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.544878    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5010ea9c-5364-4a94-b497-70dffd464e18-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:49.545149 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.544888    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5010ea9c-5364-4a94-b497-70dffd464e18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:57:49.603332 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.603294    2583 generic.go:356] "Generic (PLEG): container finished" podID="5010ea9c-5364-4a94-b497-70dffd464e18" containerID="3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5" exitCode=0
Sep 18 19:57:49.603534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.603396    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"
Sep 18 19:57:49.603534 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.603393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerDied","Data":"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"}
Sep 18 19:57:49.603666 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.603547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph" event={"ID":"5010ea9c-5364-4a94-b497-70dffd464e18","Type":"ContainerDied","Data":"f7f7a4006c44a6e94b79071fa73e4269a25f7d0e272d6c0d0d3180bce613ee4c"}
Sep 18 19:57:49.603666 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.603589    2583 scope.go:117] "RemoveContainer" containerID="297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"
Sep 18 19:57:49.612294 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.612184    2583 scope.go:117] "RemoveContainer" containerID="3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"
Sep 18 19:57:49.620108 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.620091    2583 scope.go:117] "RemoveContainer" containerID="c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"
Sep 18 19:57:49.627045 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.627016    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:49.628193 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.628175    2583 scope.go:117] "RemoveContainer" containerID="297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"
Sep 18 19:57:49.628491 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:49.628471    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4\": container with ID starting with 297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4 not found: ID does not exist" containerID="297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"
Sep 18 19:57:49.628547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.628504    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4"} err="failed to get container status \"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4\": rpc error: code = NotFound desc = could not find container \"297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4\": container with ID starting with 297f01a5836ecdb34c7b55d87cf2035dda91fb15b4deb3f972cf2e57a849cfb4 not found: ID does not exist"
Sep 18 19:57:49.628547 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.628529    2583 scope.go:117] "RemoveContainer" containerID="3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"
Sep 18 19:57:49.628806 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:49.628788    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5\": container with ID starting with 3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5 not found: ID does not exist" containerID="3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"
Sep 18 19:57:49.628879 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.628818    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5"} err="failed to get container status \"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5\": rpc error: code = NotFound desc = could not find container \"3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5\": container with ID starting with 3f078f9a4f575d7e42b1bf29860ac81da0f85141d995c9d4ce290ee62ecb80d5 not found: ID does not exist"
Sep 18 19:57:49.628879 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.628841    2583 scope.go:117] "RemoveContainer" containerID="c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"
Sep 18 19:57:49.629200 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:57:49.629170    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352\": container with ID starting with c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352 not found: ID does not exist" containerID="c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"
Sep 18 19:57:49.629303 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.629214    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352"} err="failed to get container status \"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352\": rpc error: code = NotFound desc = could not find container \"c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352\": container with ID starting with c89d92336f9c1cfe75371dbc0e5c0855dbed0184ce1c40452d64087349b32352 not found: ID does not exist"
Sep 18 19:57:49.631596 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.631575    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-544db88986-nsbph"]
Sep 18 19:57:49.791305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:49.791263    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" path="/var/lib/kubelet/pods/5010ea9c-5364-4a94-b497-70dffd464e18/volumes"
Sep 18 19:57:53.281985 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.281948    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282291    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="storage-initializer"
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282302    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="storage-initializer"
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282318    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kube-rbac-proxy"
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282325    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kube-rbac-proxy"
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282341    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container"
Sep 18 19:57:53.282410 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container"
Sep 18 19:57:53.282613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282426    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kserve-container"
Sep 18 19:57:53.282613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.282438    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5010ea9c-5364-4a94-b497-70dffd464e18" containerName="kube-rbac-proxy"
Sep 18 19:57:53.287196 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.287176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.290022 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.289997    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 18 19:57:53.290324 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.290304    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 18 19:57:53.298720 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.298694    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:57:53.377725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.377681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.377725 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.377730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.378002 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.377749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.378002 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.377868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhxl\" (UniqueName: \"kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.378002 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.377922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479071 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhxl\" (UniqueName: \"kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479071 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479357 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479580 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479866 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.479946 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.479924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.481810 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.481786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.491181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.491156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhxl\" (UniqueName: \"kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl\") pod \"isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.598438 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.598335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:53.892538 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:53.892386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:57:53.894868 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:57:53.894839    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8253af27_eed9_4caa_b2a2_87b15d7b823f.slice/crio-a4e1a07dcb90ce0348d0b0289639c1c11ffc2894375d90125ac175369f172a52 WatchSource:0}: Error finding container a4e1a07dcb90ce0348d0b0289639c1c11ffc2894375d90125ac175369f172a52: Status 404 returned error can't find the container with id a4e1a07dcb90ce0348d0b0289639c1c11ffc2894375d90125ac175369f172a52
Sep 18 19:57:54.626187 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:54.626149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerStarted","Data":"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"}
Sep 18 19:57:54.626666 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:54.626208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerStarted","Data":"a4e1a07dcb90ce0348d0b0289639c1c11ffc2894375d90125ac175369f172a52"}
Sep 18 19:57:56.487555 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:56.487507    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:57:58.644137 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:58.644105    2583 generic.go:356] "Generic (PLEG): container finished" podID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerID="d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3" exitCode=0
Sep 18 19:57:58.644658 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:58.644159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerDied","Data":"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"}
Sep 18 19:57:59.650727 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:59.650687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerStarted","Data":"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"}
Sep 18 19:57:59.651113 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:59.650769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerStarted","Data":"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"}
Sep 18 19:57:59.651113 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:59.650987    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:57:59.685625 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:57:59.685560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" podStartSLOduration=2.611915769 podStartE2EDuration="6.685541706s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.073625937s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:59.684064576 +0000 UTC m=+2612.059490871" watchObservedRunningTime="2026-09-18 19:57:59.685541706 +0000 UTC m=+2612.060967999"
Sep 18 19:58:00.655735 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:00.655697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:58:06.487443 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:06.487402    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:58:06.666475 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:06.666438    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:58:16.487703 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:16.487662    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:58:26.487900 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:26.487850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:58:36.488150 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:36.488118    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:58:36.670692 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:36.670661    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:58:43.322081 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:43.322045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:58:43.322496 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:43.322270    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kserve-container" containerID="cri-o://70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23" gracePeriod=30
Sep 18 19:58:43.322496 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:43.322321    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kube-rbac-proxy" containerID="cri-o://9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43" gracePeriod=30
Sep 18 19:58:43.848318 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:43.848284    2583 generic.go:356] "Generic (PLEG): container finished" podID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerID="9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43" exitCode=2
Sep 18 19:58:43.848496 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:43.848351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerDied","Data":"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"}
Sep 18 19:58:44.575139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.575110    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:58:44.608445 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location\") pod \"8253af27-eed9-4caa-b2a2-87b15d7b823f\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") "
Sep 18 19:58:44.608618 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert\") pod \"8253af27-eed9-4caa-b2a2-87b15d7b823f\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") "
Sep 18 19:58:44.608618 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls\") pod \"8253af27-eed9-4caa-b2a2-87b15d7b823f\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") "
Sep 18 19:58:44.608746 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608665    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fhxl\" (UniqueName: \"kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl\") pod \"8253af27-eed9-4caa-b2a2-87b15d7b823f\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") "
Sep 18 19:58:44.608746 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608717    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config\") pod \"8253af27-eed9-4caa-b2a2-87b15d7b823f\" (UID: \"8253af27-eed9-4caa-b2a2-87b15d7b823f\") "
Sep 18 19:58:44.608864 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8253af27-eed9-4caa-b2a2-87b15d7b823f" (UID: "8253af27-eed9-4caa-b2a2-87b15d7b823f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:44.608939 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.608906    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8253af27-eed9-4caa-b2a2-87b15d7b823f" (UID: "8253af27-eed9-4caa-b2a2-87b15d7b823f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:44.609117 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.609096    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:44.609194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.609120    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8253af27-eed9-4caa-b2a2-87b15d7b823f-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:44.609194 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.609123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8253af27-eed9-4caa-b2a2-87b15d7b823f" (UID: "8253af27-eed9-4caa-b2a2-87b15d7b823f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:44.610662 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.610640    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8253af27-eed9-4caa-b2a2-87b15d7b823f" (UID: "8253af27-eed9-4caa-b2a2-87b15d7b823f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:44.611187 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.611166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl" (OuterVolumeSpecName: "kube-api-access-8fhxl") pod "8253af27-eed9-4caa-b2a2-87b15d7b823f" (UID: "8253af27-eed9-4caa-b2a2-87b15d7b823f"). InnerVolumeSpecName "kube-api-access-8fhxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:44.710139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.710056    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8253af27-eed9-4caa-b2a2-87b15d7b823f-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:44.710139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.710087    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhxl\" (UniqueName: \"kubernetes.io/projected/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-api-access-8fhxl\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:44.710139 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.710096    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8253af27-eed9-4caa-b2a2-87b15d7b823f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:44.854656 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.854621    2583 generic.go:356] "Generic (PLEG): container finished" podID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerID="70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23" exitCode=0
Sep 18 19:58:44.854831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.854673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerDied","Data":"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"}
Sep 18 19:58:44.854831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.854682    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"
Sep 18 19:58:44.854831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.854726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh" event={"ID":"8253af27-eed9-4caa-b2a2-87b15d7b823f","Type":"ContainerDied","Data":"a4e1a07dcb90ce0348d0b0289639c1c11ffc2894375d90125ac175369f172a52"}
Sep 18 19:58:44.854831 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.854753    2583 scope.go:117] "RemoveContainer" containerID="9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"
Sep 18 19:58:44.864154 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.864126    2583 scope.go:117] "RemoveContainer" containerID="70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"
Sep 18 19:58:44.872682 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.872664    2583 scope.go:117] "RemoveContainer" containerID="d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"
Sep 18 19:58:44.878249 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.878221    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:58:44.881608 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.881583    2583 scope.go:117] "RemoveContainer" containerID="9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"
Sep 18 19:58:44.881951 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:44.881929    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43\": container with ID starting with 9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43 not found: ID does not exist" containerID="9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"
Sep 18 19:58:44.882016 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.881960    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43"} err="failed to get container status \"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43\": rpc error: code = NotFound desc = could not find container \"9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43\": container with ID starting with 9ab3bbb78df3fb1d34bab9db4ac1c3f4021b652f6fd7fe24fc7eb92996adbf43 not found: ID does not exist"
Sep 18 19:58:44.882016 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.881980    2583 scope.go:117] "RemoveContainer" containerID="70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"
Sep 18 19:58:44.882264 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:44.882246    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23\": container with ID starting with 70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23 not found: ID does not exist" containerID="70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"
Sep 18 19:58:44.882307 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.882270    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23"} err="failed to get container status \"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23\": rpc error: code = NotFound desc = could not find container \"70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23\": container with ID starting with 70f22db944a83b8860e77b59a1373dfe9572a45d790f8fc671b98fda8b49af23 not found: ID does not exist"
Sep 18 19:58:44.882307 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.882286    2583 scope.go:117] "RemoveContainer" containerID="d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"
Sep 18 19:58:44.882456 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.882428    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-fc758d9c9-m6hjh"]
Sep 18 19:58:44.882586 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:44.882569    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3\": container with ID starting with d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3 not found: ID does not exist" containerID="d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"
Sep 18 19:58:44.882631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:44.882594    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3"} err="failed to get container status \"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3\": rpc error: code = NotFound desc = could not find container \"d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3\": container with ID starting with d4c0966eaaf53b3df4c4a13544da11e9df4bc1c68424f4ae2ef1f60724eac4b3 not found: ID does not exist"
Sep 18 19:58:45.162140 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.162109    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:58:45.162400 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.162356    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" containerID="cri-o://cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c" gracePeriod=30
Sep 18 19:58:45.162540 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.162408    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kube-rbac-proxy" containerID="cri-o://884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a" gracePeriod=30
Sep 18 19:58:45.456664 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.456561    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 18 19:58:45.791445 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.791413    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" path="/var/lib/kubelet/pods/8253af27-eed9-4caa-b2a2-87b15d7b823f/volumes"
Sep 18 19:58:45.867809 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.867777    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerID="884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a" exitCode=2
Sep 18 19:58:45.867992 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:45.867849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerDied","Data":"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"}
Sep 18 19:58:46.487336 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:46.487294    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 18 19:58:49.614019 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.613990    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:58:49.652711 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.652677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls\") pod \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") "
Sep 18 19:58:49.652904 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.652749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l52hc\" (UniqueName: \"kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc\") pod \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") "
Sep 18 19:58:49.652904 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.652783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert\") pod \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") "
Sep 18 19:58:49.652904 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.652812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config\") pod \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") "
Sep 18 19:58:49.652904 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.652870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location\") pod \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\" (UID: \"c3740fbd-7f4c-499c-a2a7-5742957aa3f3\") "
Sep 18 19:58:49.653283 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.653204    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3740fbd-7f4c-499c-a2a7-5742957aa3f3" (UID: "c3740fbd-7f4c-499c-a2a7-5742957aa3f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:49.653425 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.653226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3740fbd-7f4c-499c-a2a7-5742957aa3f3" (UID: "c3740fbd-7f4c-499c-a2a7-5742957aa3f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:49.653425 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.653375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3740fbd-7f4c-499c-a2a7-5742957aa3f3" (UID: "c3740fbd-7f4c-499c-a2a7-5742957aa3f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:49.654937 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.654905    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3740fbd-7f4c-499c-a2a7-5742957aa3f3" (UID: "c3740fbd-7f4c-499c-a2a7-5742957aa3f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:49.655032 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.654993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc" (OuterVolumeSpecName: "kube-api-access-l52hc") pod "c3740fbd-7f4c-499c-a2a7-5742957aa3f3" (UID: "c3740fbd-7f4c-499c-a2a7-5742957aa3f3"). InnerVolumeSpecName "kube-api-access-l52hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:49.753789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.753734    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:49.753789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.753780    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:49.753789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.753791    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l52hc\" (UniqueName: \"kubernetes.io/projected/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-api-access-l52hc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:49.753789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.753802    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:49.753789 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.753810    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3740fbd-7f4c-499c-a2a7-5742957aa3f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 19:58:49.889822 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.889775    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerID="cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c" exitCode=0
Sep 18 19:58:49.889997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.889838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerDied","Data":"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"}
Sep 18 19:58:49.889997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.889902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb" event={"ID":"c3740fbd-7f4c-499c-a2a7-5742957aa3f3","Type":"ContainerDied","Data":"8a501d4ff7dd80b9b44885ddbd5e2183c48355e8f0b994d14897864f59f5022d"}
Sep 18 19:58:49.889997 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.889847    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"
Sep 18 19:58:49.890119 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.889929    2583 scope.go:117] "RemoveContainer" containerID="884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"
Sep 18 19:58:49.898761 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.898741    2583 scope.go:117] "RemoveContainer" containerID="cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"
Sep 18 19:58:49.906631 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.906611    2583 scope.go:117] "RemoveContainer" containerID="6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"
Sep 18 19:58:49.915748 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.915725    2583 scope.go:117] "RemoveContainer" containerID="884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"
Sep 18 19:58:49.916126 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:49.916106    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a\": container with ID starting with 884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a not found: ID does not exist" containerID="884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"
Sep 18 19:58:49.916177 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916151    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a"} err="failed to get container status \"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a\": rpc error: code = NotFound desc = could not find container \"884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a\": container with ID starting with 884d3934f72bb49a5da29163c7a22c0f0531c06f84bb98aa4ebddb90a5d2c96a not found: ID does not exist"
Sep 18 19:58:49.916177 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916172    2583 scope.go:117] "RemoveContainer" containerID="cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"
Sep 18 19:58:49.916554 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:49.916536    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c\": container with ID starting with cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c not found: ID does not exist" containerID="cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"
Sep 18 19:58:49.916606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916548    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:58:49.916606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916557    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c"} err="failed to get container status \"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c\": rpc error: code = NotFound desc = could not find container \"cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c\": container with ID starting with cda3338f9fa6c98e582b047cbb487be14b44abd9820c494f8ecc67b03bffae1c not found: ID does not exist"
Sep 18 19:58:49.916606 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916573    2583 scope.go:117] "RemoveContainer" containerID="6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"
Sep 18 19:58:49.916826 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:49.916808    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4\": container with ID starting with 6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4 not found: ID does not exist" containerID="6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"
Sep 18 19:58:49.916885 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.916836    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4"} err="failed to get container status \"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4\": rpc error: code = NotFound desc = could not find container \"6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4\": container with ID starting with 6f111c719152ece141934be7fbad11568207172c6ea07691293776ad781be6c4 not found: ID does not exist"
Sep 18 19:58:49.921775 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:49.921754    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5c5d6c7f44-vwzqb"]
Sep 18 19:58:50.432023 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.431983    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 19:58:50.432389 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432353    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="storage-initializer"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432390    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="storage-initializer"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432403    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kube-rbac-proxy"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432413    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kube-rbac-proxy"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432422    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="storage-initializer"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432431    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="storage-initializer"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432439    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kserve-container"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432447    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kserve-container"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432469    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kube-rbac-proxy"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432477    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kube-rbac-proxy"
Sep 18 19:58:50.432493 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432495    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container"
Sep 18 19:58:50.433004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432503    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container"
Sep 18 19:58:50.433004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432589    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kube-rbac-proxy"
Sep 18 19:58:50.433004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kserve-container"
Sep 18 19:58:50.433004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432615    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8253af27-eed9-4caa-b2a2-87b15d7b823f" containerName="kube-rbac-proxy"
Sep 18 19:58:50.433004 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.432627    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" containerName="kserve-container"
Sep 18 19:58:50.437313 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.437290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.440411 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.440386    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 19:58:50.440745 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.440730    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:58:50.440806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.440762    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 18 19:58:50.441062 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.441046    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:58:50.441541 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.441525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 19:58:50.442659 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.442643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 18 19:58:50.457704 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.457668    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 19:58:50.560694 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.560660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.560893 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.560700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcld2\" (UniqueName: \"kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.560893 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.560723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.560893 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.560800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.661251 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.661207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.661259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcld2\" (UniqueName: \"kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.661291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.661345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:50.661454    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: E0918 19:58:50.661522    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls podName:61fa7c23-6534-49ff-9522-c9be950bbc61 nodeName:}" failed. No retries permitted until 2026-09-18 19:58:51.161502595 +0000 UTC m=+2663.536928881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls") pod "isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" (UID: "61fa7c23-6534-49ff-9522-c9be950bbc61") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 18 19:58:50.661783 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.661661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.662170 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.662136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:50.672613 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:50.672586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcld2\" (UniqueName: \"kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:51.165843 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.165808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:51.168275 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.168249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") pod \"isvc-sklearn-s3-predictor-78f7f6bcc9-24q46\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:51.349405 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.349356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:51.626622 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.626592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 19:58:51.628692 ip-10-0-135-205 kubenswrapper[2583]: W0918 19:58:51.628662    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fa7c23_6534_49ff_9522_c9be950bbc61.slice/crio-5af0bf866c685dd39a9d8fbd81d926ce412f0910c3006d82f271bd73125e4ed2 WatchSource:0}: Error finding container 5af0bf866c685dd39a9d8fbd81d926ce412f0910c3006d82f271bd73125e4ed2: Status 404 returned error can't find the container with id 5af0bf866c685dd39a9d8fbd81d926ce412f0910c3006d82f271bd73125e4ed2
Sep 18 19:58:51.792691 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.792661    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3740fbd-7f4c-499c-a2a7-5742957aa3f3" path="/var/lib/kubelet/pods/c3740fbd-7f4c-499c-a2a7-5742957aa3f3/volumes"
Sep 18 19:58:51.901006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.900918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerStarted","Data":"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"}
Sep 18 19:58:51.901006 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:51.900988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerStarted","Data":"5af0bf866c685dd39a9d8fbd81d926ce412f0910c3006d82f271bd73125e4ed2"}
Sep 18 19:58:53.911048 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:53.911014    2583 generic.go:356] "Generic (PLEG): container finished" podID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerID="3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f" exitCode=0
Sep 18 19:58:53.911473 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:53.911089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerDied","Data":"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"}
Sep 18 19:58:54.918653 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:54.918622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerStarted","Data":"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"}
Sep 18 19:58:54.919091 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:54.918682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerStarted","Data":"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"}
Sep 18 19:58:54.919091 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:54.918860    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:54.941515 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:54.941462    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podStartSLOduration=3.736958394 podStartE2EDuration="4.941445463s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.204487069s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:58:54.939965087 +0000 UTC m=+2667.315391381" watchObservedRunningTime="2026-09-18 19:58:54.941445463 +0000 UTC m=+2667.316871774"
Sep 18 19:58:55.923195 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:55.923163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:58:55.924458 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:55.924425    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:58:56.927806 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:58:56.927767    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:01.932863 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:01.932833    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 19:59:01.933489 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:01.933460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:11.934181 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:11.934094    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:21.934305 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:21.934259    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:31.934075 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:31.934029    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:41.933449 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:41.933410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 19:59:51.933454 ip-10-0-135-205 kubenswrapper[2583]: I0918 19:59:51.933411    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:00:01.933644 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:01.933601    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:00:02.786873 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:02.786833    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:00:12.786901 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:12.786852    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:00:13.863862 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.863827    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:00:13.867417 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.867397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:13.870592 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.870552    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:00:13.870592 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.870560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:00:13.870797 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.870597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 18 20:00:13.870797 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.870602    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 18 20:00:13.889351 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.889320    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:00:13.989903 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.989869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:13.990098 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.989928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:13.990098 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.990001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:13.990098 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.990057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvk5p\" (UniqueName: \"kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:13.990203 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:13.990096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.090636 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.090598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.090847 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.090658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.090847 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.090717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.090847 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.090733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvk5p\" (UniqueName: \"kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.090847 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.090758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.091178 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.091155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.091338 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.091318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.091420 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.091404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.093388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.093353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.102645 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.102623    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvk5p\" (UniqueName: \"kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p\") pod \"isvc-paddle-predictor-69579d8675-rc888\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.177909 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.177793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:14.458484 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:14.458457    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:00:14.460547 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:00:14.460515    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2593e3ae_c043_462a_85df_184757c79362.slice/crio-85a8732d618450b3ac343ecb5e1917f04d447bfedff2b63187451dc5ef3e5cb7 WatchSource:0}: Error finding container 85a8732d618450b3ac343ecb5e1917f04d447bfedff2b63187451dc5ef3e5cb7: Status 404 returned error can't find the container with id 85a8732d618450b3ac343ecb5e1917f04d447bfedff2b63187451dc5ef3e5cb7
Sep 18 20:00:15.271101 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:15.271060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerStarted","Data":"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"}
Sep 18 20:00:15.271540 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:15.271124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerStarted","Data":"85a8732d618450b3ac343ecb5e1917f04d447bfedff2b63187451dc5ef3e5cb7"}
Sep 18 20:00:20.297073 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:20.297041    2583 generic.go:356] "Generic (PLEG): container finished" podID="2593e3ae-c043-462a-85df-184757c79362" containerID="478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641" exitCode=0
Sep 18 20:00:20.297512 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:20.297109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerDied","Data":"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"}
Sep 18 20:00:20.298518 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:20.298497    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:00:22.788559 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:22.788520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 20:00:31.187805 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.187761    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 20:00:31.215826 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.188039    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" containerID="cri-o://f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179" gracePeriod=30
Sep 18 20:00:31.215826 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.188052    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kube-rbac-proxy" containerID="cri-o://0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9" gracePeriod=30
Sep 18 20:00:31.366592 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.366553    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:00:31.373942 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.373913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.379076 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.379042    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 18 20:00:31.379259 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.379218    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:00:31.389561 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.389505    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:00:31.432969 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.432932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdr2\" (UniqueName: \"kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.433165 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.432988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.433165 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.433048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.433165 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.433079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.433165 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.433129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534234 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534463 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534463 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534603 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534658 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdr2\" (UniqueName: \"kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.534749 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.534719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.535232 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.535209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.535341 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.535316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.537103 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.537079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.544175 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.544129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdr2\" (UniqueName: \"kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.685726 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.685682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:31.855871 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.855838    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:00:31.856468 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.856446    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:00:31.928212 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:31.928178    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 18 20:00:32.111310 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.111278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:00:32.112357 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:00:32.112333    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3d944d_4beb_424d_a5f7_33aac5bf845d.slice/crio-fa7f87cfbbd0baf33170cb5d6c269f9551a239608e39fa6df3fa8c28881acc80 WatchSource:0}: Error finding container fa7f87cfbbd0baf33170cb5d6c269f9551a239608e39fa6df3fa8c28881acc80: Status 404 returned error can't find the container with id fa7f87cfbbd0baf33170cb5d6c269f9551a239608e39fa6df3fa8c28881acc80
Sep 18 20:00:32.362580 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.362488    2583 generic.go:356] "Generic (PLEG): container finished" podID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerID="0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9" exitCode=2
Sep 18 20:00:32.362580 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.362560    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerDied","Data":"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"}
Sep 18 20:00:32.365000 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.364971    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerStarted","Data":"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"}
Sep 18 20:00:32.365167 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.365041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerStarted","Data":"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"}
Sep 18 20:00:32.365321 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.365293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:32.366741 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.366719    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerStarted","Data":"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"}
Sep 18 20:00:32.366861 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.366774    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerStarted","Data":"fa7f87cfbbd0baf33170cb5d6c269f9551a239608e39fa6df3fa8c28881acc80"}
Sep 18 20:00:32.393238 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.387321    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podStartSLOduration=2.648048633 podStartE2EDuration="19.387276682s" totalImagesPullingTime="11.594354695s" totalInitContainerRuntime="5.144873354s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:32.386315858 +0000 UTC m=+2764.761742164" watchObservedRunningTime="2026-09-18 20:00:32.387276682 +0000 UTC m=+2764.762702978"
Sep 18 20:00:32.787013 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:32.786969    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 18 20:00:33.372868 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:33.372833    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerID="9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034" exitCode=0
Sep 18 20:00:33.373334 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:33.372908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerDied","Data":"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"}
Sep 18 20:00:33.373334 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:33.373246    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:33.374772 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:33.374748    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:00:34.379279 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.379242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerStarted","Data":"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"}
Sep 18 20:00:34.379771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.379311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerStarted","Data":"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"}
Sep 18 20:00:34.379771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.379534    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:34.379771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.379577    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:34.379771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.379569    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:00:34.380906 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.380880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:00:34.403310 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:34.403232    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podStartSLOduration=2.277621163 podStartE2EDuration="3.40320831s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.125587147s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:34.401549777 +0000 UTC m=+2766.776976071" watchObservedRunningTime="2026-09-18 20:00:34.40320831 +0000 UTC m=+2766.778634606"
Sep 18 20:00:35.383779 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:35.383734    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:00:36.738460 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.738430    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 20:00:36.780907 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.780802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location\") pod \"61fa7c23-6534-49ff-9522-c9be950bbc61\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") "
Sep 18 20:00:36.780907 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.780889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config\") pod \"61fa7c23-6534-49ff-9522-c9be950bbc61\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") "
Sep 18 20:00:36.781169 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.780925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcld2\" (UniqueName: \"kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2\") pod \"61fa7c23-6534-49ff-9522-c9be950bbc61\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") "
Sep 18 20:00:36.781169 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.780971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") pod \"61fa7c23-6534-49ff-9522-c9be950bbc61\" (UID: \"61fa7c23-6534-49ff-9522-c9be950bbc61\") "
Sep 18 20:00:36.781300 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.781171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61fa7c23-6534-49ff-9522-c9be950bbc61" (UID: "61fa7c23-6534-49ff-9522-c9be950bbc61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:36.781300 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.781250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61fa7c23-6534-49ff-9522-c9be950bbc61" (UID: "61fa7c23-6534-49ff-9522-c9be950bbc61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:36.783098 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.783070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2" (OuterVolumeSpecName: "kube-api-access-xcld2") pod "61fa7c23-6534-49ff-9522-c9be950bbc61" (UID: "61fa7c23-6534-49ff-9522-c9be950bbc61"). InnerVolumeSpecName "kube-api-access-xcld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:36.783236 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.783174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61fa7c23-6534-49ff-9522-c9be950bbc61" (UID: "61fa7c23-6534-49ff-9522-c9be950bbc61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:36.881940 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.881898    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:00:36.881940 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.881936    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcld2\" (UniqueName: \"kubernetes.io/projected/61fa7c23-6534-49ff-9522-c9be950bbc61-kube-api-access-xcld2\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:00:36.881940 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.881947    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61fa7c23-6534-49ff-9522-c9be950bbc61-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:00:36.882424 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:36.881955    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61fa7c23-6534-49ff-9522-c9be950bbc61-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:00:37.395744 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.395711    2583 generic.go:356] "Generic (PLEG): container finished" podID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerID="f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179" exitCode=0
Sep 18 20:00:37.395958 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.395781    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"
Sep 18 20:00:37.395958 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.395798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerDied","Data":"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"}
Sep 18 20:00:37.395958 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.395859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46" event={"ID":"61fa7c23-6534-49ff-9522-c9be950bbc61","Type":"ContainerDied","Data":"5af0bf866c685dd39a9d8fbd81d926ce412f0910c3006d82f271bd73125e4ed2"}
Sep 18 20:00:37.395958 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.395885    2583 scope.go:117] "RemoveContainer" containerID="0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"
Sep 18 20:00:37.413642 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.413619    2583 scope.go:117] "RemoveContainer" containerID="f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"
Sep 18 20:00:37.421619 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.421594    2583 scope.go:117] "RemoveContainer" containerID="3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"
Sep 18 20:00:37.426204 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.426176    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 20:00:37.430228 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.430207    2583 scope.go:117] "RemoveContainer" containerID="0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"
Sep 18 20:00:37.430720 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:00:37.430688    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9\": container with ID starting with 0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9 not found: ID does not exist" containerID="0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"
Sep 18 20:00:37.430846 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.430757    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9"} err="failed to get container status \"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9\": rpc error: code = NotFound desc = could not find container \"0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9\": container with ID starting with 0e905a2c496b9a109d1e7ffcbe217b372394aedf8b1d2c827a58ea1bf68b87e9 not found: ID does not exist"
Sep 18 20:00:37.430846 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.430793    2583 scope.go:117] "RemoveContainer" containerID="f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"
Sep 18 20:00:37.431227 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:00:37.431201    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179\": container with ID starting with f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179 not found: ID does not exist" containerID="f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"
Sep 18 20:00:37.431401 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.431342    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179"} err="failed to get container status \"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179\": rpc error: code = NotFound desc = could not find container \"f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179\": container with ID starting with f3a68d81f4012fc3ed9f6d8b7b58ac6cd44a20cbcb66a7bdc469b2d3331d4179 not found: ID does not exist"
Sep 18 20:00:37.431401 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.431398    2583 scope.go:117] "RemoveContainer" containerID="3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"
Sep 18 20:00:37.431752 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:00:37.431732    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f\": container with ID starting with 3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f not found: ID does not exist" containerID="3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"
Sep 18 20:00:37.431840 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.431756    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f"} err="failed to get container status \"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f\": rpc error: code = NotFound desc = could not find container \"3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f\": container with ID starting with 3f6225bfd7fcd1d01d246a629a8d7cd52f9c6ca49138c9780247ae4125868a5f not found: ID does not exist"
Sep 18 20:00:37.432844 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.432824    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-78f7f6bcc9-24q46"]
Sep 18 20:00:37.791811 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:37.791776    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" path="/var/lib/kubelet/pods/61fa7c23-6534-49ff-9522-c9be950bbc61/volumes"
Sep 18 20:00:39.384072 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:39.384042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:00:39.384625 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:39.384601    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:00:40.388797 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:40.388724    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:00:40.389249 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:40.389215    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:00:49.385164 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:49.385124    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:00:50.389274 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:50.389232    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:00:59.384887 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:00:59.384844    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:01:00.390083 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:00.390043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:09.385379 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:09.385333    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:01:10.389840 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:10.389799    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:19.385207 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:19.385163    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 18 20:01:20.390169 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:20.390129    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:29.385742 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:29.385710    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:01:30.389957 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:30.389914    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:35.155285 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:35.155244    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:01:35.155838 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:35.155600    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container" containerID="cri-o://c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd" gracePeriod=30
Sep 18 20:01:35.155838 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:35.155653    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kube-rbac-proxy" containerID="cri-o://d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940" gracePeriod=30
Sep 18 20:01:35.664224 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:35.664190    2583 generic.go:356] "Generic (PLEG): container finished" podID="2593e3ae-c043-462a-85df-184757c79362" containerID="d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940" exitCode=2
Sep 18 20:01:35.664434 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:35.664262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerDied","Data":"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"}
Sep 18 20:01:38.509180 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.509154    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:01:38.614262 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls\") pod \"2593e3ae-c043-462a-85df-184757c79362\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") "
Sep 18 20:01:38.614262 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614243    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvk5p\" (UniqueName: \"kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p\") pod \"2593e3ae-c043-462a-85df-184757c79362\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") "
Sep 18 20:01:38.614507 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614268    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config\") pod \"2593e3ae-c043-462a-85df-184757c79362\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") "
Sep 18 20:01:38.614507 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614287    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert\") pod \"2593e3ae-c043-462a-85df-184757c79362\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") "
Sep 18 20:01:38.614507 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location\") pod \"2593e3ae-c043-462a-85df-184757c79362\" (UID: \"2593e3ae-c043-462a-85df-184757c79362\") "
Sep 18 20:01:38.614773 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2593e3ae-c043-462a-85df-184757c79362" (UID: "2593e3ae-c043-462a-85df-184757c79362"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:38.614827 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.614749    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2593e3ae-c043-462a-85df-184757c79362" (UID: "2593e3ae-c043-462a-85df-184757c79362"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:38.616454 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.616418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p" (OuterVolumeSpecName: "kube-api-access-rvk5p") pod "2593e3ae-c043-462a-85df-184757c79362" (UID: "2593e3ae-c043-462a-85df-184757c79362"). InnerVolumeSpecName "kube-api-access-rvk5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:38.616454 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.616430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2593e3ae-c043-462a-85df-184757c79362" (UID: "2593e3ae-c043-462a-85df-184757c79362"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:38.623801 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.623768    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2593e3ae-c043-462a-85df-184757c79362" (UID: "2593e3ae-c043-462a-85df-184757c79362"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:38.679290 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.679253    2583 generic.go:356] "Generic (PLEG): container finished" podID="2593e3ae-c043-462a-85df-184757c79362" containerID="c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd" exitCode=0
Sep 18 20:01:38.679520 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.679300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerDied","Data":"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"}
Sep 18 20:01:38.679520 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.679325    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"
Sep 18 20:01:38.679520 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.679357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888" event={"ID":"2593e3ae-c043-462a-85df-184757c79362","Type":"ContainerDied","Data":"85a8732d618450b3ac343ecb5e1917f04d447bfedff2b63187451dc5ef3e5cb7"}
Sep 18 20:01:38.679520 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.679416    2583 scope.go:117] "RemoveContainer" containerID="d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"
Sep 18 20:01:38.688747 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.688721    2583 scope.go:117] "RemoveContainer" containerID="c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"
Sep 18 20:01:38.696656 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.696636    2583 scope.go:117] "RemoveContainer" containerID="478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"
Sep 18 20:01:38.704997 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.704963    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:01:38.705355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.705333    2583 scope.go:117] "RemoveContainer" containerID="d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"
Sep 18 20:01:38.705801 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:01:38.705780    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940\": container with ID starting with d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940 not found: ID does not exist" containerID="d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"
Sep 18 20:01:38.705878 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.705816    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940"} err="failed to get container status \"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940\": rpc error: code = NotFound desc = could not find container \"d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940\": container with ID starting with d2d750465252ff8798052ee3c5299ac2eb36c6b0e25986c0231a147f9c1ac940 not found: ID does not exist"
Sep 18 20:01:38.705878 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.705863    2583 scope.go:117] "RemoveContainer" containerID="c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"
Sep 18 20:01:38.706202 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:01:38.706183    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd\": container with ID starting with c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd not found: ID does not exist" containerID="c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"
Sep 18 20:01:38.706256 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.706211    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd"} err="failed to get container status \"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd\": rpc error: code = NotFound desc = could not find container \"c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd\": container with ID starting with c4cc90e85c75bf6403e9d9b1b542db8f311629c746dd11ea4df0cc7dc4b5d1cd not found: ID does not exist"
Sep 18 20:01:38.706256 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.706230    2583 scope.go:117] "RemoveContainer" containerID="478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"
Sep 18 20:01:38.706565 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:01:38.706541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641\": container with ID starting with 478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641 not found: ID does not exist" containerID="478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"
Sep 18 20:01:38.706619 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.706571    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641"} err="failed to get container status \"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641\": rpc error: code = NotFound desc = could not find container \"478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641\": container with ID starting with 478864570ada5506decd4ec24e0f99b81af3a7feb4b5103a872e3c65d2a07641 not found: ID does not exist"
Sep 18 20:01:38.710821 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.710783    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-69579d8675-rc888"]
Sep 18 20:01:38.715379 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.715345    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2593e3ae-c043-462a-85df-184757c79362-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:01:38.715452 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.715381    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvk5p\" (UniqueName: \"kubernetes.io/projected/2593e3ae-c043-462a-85df-184757c79362-kube-api-access-rvk5p\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:01:38.715452 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.715397    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:01:38.715452 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.715406    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2593e3ae-c043-462a-85df-184757c79362-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:01:38.715452 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:38.715415    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2593e3ae-c043-462a-85df-184757c79362-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:01:39.792257 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:39.792221    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2593e3ae-c043-462a-85df-184757c79362" path="/var/lib/kubelet/pods/2593e3ae-c043-462a-85df-184757c79362/volumes"
Sep 18 20:01:40.389327 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.389274    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:40.442556 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442517    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:01:40.442904 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442889    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="storage-initializer"
Sep 18 20:01:40.442904 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442905    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="storage-initializer"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442919    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442925    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442940    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kube-rbac-proxy"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442966    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kube-rbac-proxy"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442979    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kube-rbac-proxy"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kube-rbac-proxy"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442992    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="storage-initializer"
Sep 18 20:01:40.443016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.442997    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="storage-initializer"
Sep 18 20:01:40.443375 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.443073    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kube-rbac-proxy"
Sep 18 20:01:40.443375 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.443080    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kserve-container"
Sep 18 20:01:40.443375 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.443088    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fa7c23-6534-49ff-9522-c9be950bbc61" containerName="kserve-container"
Sep 18 20:01:40.443375 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.443095    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2593e3ae-c043-462a-85df-184757c79362" containerName="kube-rbac-proxy"
Sep 18 20:01:40.446350 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.446331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.449640 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.449609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 18 20:01:40.449640 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.449614    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:01:40.449823 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.449618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:01:40.458433 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.458403    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:01:40.530244 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.530212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.530476 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.530266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.530476 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.530339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.530476 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.530454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfddc\" (UniqueName: \"kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.530641 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.530523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.631830 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.631775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632051 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.631867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632051 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.631897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfddc\" (UniqueName: \"kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632051 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.631958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632051 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.631999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632297 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:01:40.632133    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 18 20:01:40.632297 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:01:40.632193    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls podName:f435e44a-acbb-4e83-9fe8-322e7773a41a nodeName:}" failed. No retries permitted until 2026-09-18 20:01:41.132175333 +0000 UTC m=+2833.507601604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls") pod "isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 18 20:01:40.632466 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.632445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632689 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.632669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.632789 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.632769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:40.643348 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:40.643286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfddc\" (UniqueName: \"kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:41.137908 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:41.137867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:41.140447 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:41.140421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:41.356109 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:41.356068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:41.633572 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:41.633547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:01:41.635518 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:01:41.635491    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf435e44a_acbb_4e83_9fe8_322e7773a41a.slice/crio-78901ac77c085980d0784510d588c6c2f1d141d5a0d3a3771e968c295de01ede WatchSource:0}: Error finding container 78901ac77c085980d0784510d588c6c2f1d141d5a0d3a3771e968c295de01ede: Status 404 returned error can't find the container with id 78901ac77c085980d0784510d588c6c2f1d141d5a0d3a3771e968c295de01ede
Sep 18 20:01:41.696475 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:41.696445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerStarted","Data":"78901ac77c085980d0784510d588c6c2f1d141d5a0d3a3771e968c295de01ede"}
Sep 18 20:01:42.702265 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:42.702227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerStarted","Data":"f62bac3ea39c79cb13954ea0ca3aa27372f69e51f8ba248ea6493f07eca21dd5"}
Sep 18 20:01:46.720496 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:46.720457    2583 generic.go:356] "Generic (PLEG): container finished" podID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerID="f62bac3ea39c79cb13954ea0ca3aa27372f69e51f8ba248ea6493f07eca21dd5" exitCode=0
Sep 18 20:01:46.720902 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:46.720541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerDied","Data":"f62bac3ea39c79cb13954ea0ca3aa27372f69e51f8ba248ea6493f07eca21dd5"}
Sep 18 20:01:47.727506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:47.727468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerStarted","Data":"2db5ed48ccc2ff5dbb8c80f8543c8c9bbdf21a89201f6bfb0311cdd18b1d0b16"}
Sep 18 20:01:47.727897 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:47.727539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerStarted","Data":"b62f77e52ae858db53c747dde579e71414b17f2ea250d27712e7b37f5e01b450"}
Sep 18 20:01:47.727897 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:47.727731    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:47.754296 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:47.754239    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podStartSLOduration=3.26906502 podStartE2EDuration="7.754220641s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.485155621s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:47.751973227 +0000 UTC m=+2840.127399520" watchObservedRunningTime="2026-09-18 20:01:47.754220641 +0000 UTC m=+2840.129646933"
Sep 18 20:01:48.732468 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:48.732385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:48.733842 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:48.733810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:01:49.737060 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:49.737019    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:01:50.390064 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:50.390025    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 18 20:01:54.741828 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:54.741796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:01:54.742411 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:01:54.742384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:00.390529 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:00.390496    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:02:01.404379 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:01.404326    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:02:01.404817 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:01.404600    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container" containerID="cri-o://c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c" gracePeriod=30
Sep 18 20:02:01.404817 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:01.404646    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kube-rbac-proxy" containerID="cri-o://b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b" gracePeriod=30
Sep 18 20:02:01.790686 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:01.790649    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerID="b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b" exitCode=2
Sep 18 20:02:01.790945 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:01.790921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerDied","Data":"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"}
Sep 18 20:02:04.742336 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:04.742295    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:05.384387 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:05.384318    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 18 20:02:06.752126 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.752098    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:02:06.765958 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.765919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert\") pod \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") "
Sep 18 20:02:06.766171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.765969    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location\") pod \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") "
Sep 18 20:02:06.766171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.765989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls\") pod \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") "
Sep 18 20:02:06.766171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.766060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sdr2\" (UniqueName: \"kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2\") pod \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") "
Sep 18 20:02:06.766171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.766103    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config\") pod \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\" (UID: \"1c3d944d-4beb-424d-a5f7-33aac5bf845d\") "
Sep 18 20:02:06.766408 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.766298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c3d944d-4beb-424d-a5f7-33aac5bf845d" (UID: "1c3d944d-4beb-424d-a5f7-33aac5bf845d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:06.766408 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.766351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c3d944d-4beb-424d-a5f7-33aac5bf845d" (UID: "1c3d944d-4beb-424d-a5f7-33aac5bf845d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:06.766583 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.766557    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c3d944d-4beb-424d-a5f7-33aac5bf845d" (UID: "1c3d944d-4beb-424d-a5f7-33aac5bf845d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:06.768317 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.768285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2" (OuterVolumeSpecName: "kube-api-access-8sdr2") pod "1c3d944d-4beb-424d-a5f7-33aac5bf845d" (UID: "1c3d944d-4beb-424d-a5f7-33aac5bf845d"). InnerVolumeSpecName "kube-api-access-8sdr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:06.768460 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.768324    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c3d944d-4beb-424d-a5f7-33aac5bf845d" (UID: "1c3d944d-4beb-424d-a5f7-33aac5bf845d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:06.824252 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.824214    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerID="c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c" exitCode=0
Sep 18 20:02:06.824445 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.824315    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"
Sep 18 20:02:06.824445 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.824320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerDied","Data":"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"}
Sep 18 20:02:06.824548 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.824442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t" event={"ID":"1c3d944d-4beb-424d-a5f7-33aac5bf845d","Type":"ContainerDied","Data":"fa7f87cfbbd0baf33170cb5d6c269f9551a239608e39fa6df3fa8c28881acc80"}
Sep 18 20:02:06.824548 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.824485    2583 scope.go:117] "RemoveContainer" containerID="b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"
Sep 18 20:02:06.834303 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.834283    2583 scope.go:117] "RemoveContainer" containerID="c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"
Sep 18 20:02:06.842983 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.842960    2583 scope.go:117] "RemoveContainer" containerID="9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"
Sep 18 20:02:06.849824 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.849793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:02:06.854560 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.854521    2583 scope.go:117] "RemoveContainer" containerID="b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"
Sep 18 20:02:06.855312 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:02:06.855289    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b\": container with ID starting with b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b not found: ID does not exist" containerID="b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"
Sep 18 20:02:06.855447 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.855319    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b"} err="failed to get container status \"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b\": rpc error: code = NotFound desc = could not find container \"b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b\": container with ID starting with b0e9e4a196bbdfced51b8485434eb922561b8ce33345ebaa029b0158409e6a0b not found: ID does not exist"
Sep 18 20:02:06.855447 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.855345    2583 scope.go:117] "RemoveContainer" containerID="c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"
Sep 18 20:02:06.855726 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:02:06.855695    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c\": container with ID starting with c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c not found: ID does not exist" containerID="c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"
Sep 18 20:02:06.855825 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.855731    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c"} err="failed to get container status \"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c\": rpc error: code = NotFound desc = could not find container \"c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c\": container with ID starting with c8f78a58850ebddb156ccecad8cc5bfb03426e44fcd6ee0d334d25a776cc2d2c not found: ID does not exist"
Sep 18 20:02:06.855825 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.855756    2583 scope.go:117] "RemoveContainer" containerID="9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"
Sep 18 20:02:06.856123 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:02:06.856104    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034\": container with ID starting with 9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034 not found: ID does not exist" containerID="9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"
Sep 18 20:02:06.856194 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.856130    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034"} err="failed to get container status \"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034\": rpc error: code = NotFound desc = could not find container \"9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034\": container with ID starting with 9271d5d31bcb72bdd652f41083c638b25a2626fff9eb3cca013d0953cf28e034 not found: ID does not exist"
Sep 18 20:02:06.857355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.857330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-688c74c647-tps4t"]
Sep 18 20:02:06.866634 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.866606    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdr2\" (UniqueName: \"kubernetes.io/projected/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-api-access-8sdr2\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:06.866634 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.866632    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:06.866778 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.866642    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c3d944d-4beb-424d-a5f7-33aac5bf845d-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:06.866778 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.866651    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c3d944d-4beb-424d-a5f7-33aac5bf845d-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:06.866778 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:06.866660    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c3d944d-4beb-424d-a5f7-33aac5bf845d-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:07.792204 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:07.792167    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" path="/var/lib/kubelet/pods/1c3d944d-4beb-424d-a5f7-33aac5bf845d/volumes"
Sep 18 20:02:13.675492 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675409    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675787    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="storage-initializer"
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675800    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="storage-initializer"
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675825    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container"
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675831    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container"
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675838    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kube-rbac-proxy"
Sep 18 20:02:13.675866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675844    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kube-rbac-proxy"
Sep 18 20:02:13.676055 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kserve-container"
Sep 18 20:02:13.676055 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.675906    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3d944d-4beb-424d-a5f7-33aac5bf845d" containerName="kube-rbac-proxy"
Sep 18 20:02:13.679057 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.679038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.682017 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.681993    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 18 20:02:13.682017 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.682011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:02:13.682153 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.682024    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:02:13.691208 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.691183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:02:13.715433 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.715396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msds7\" (UniqueName: \"kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.715433 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.715437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.715677 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.715566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.715677 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.715605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.715677 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.715662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.816651 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.816609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.816859 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.816679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msds7\" (UniqueName: \"kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.816859 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.816712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.816859 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.816773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.816859 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.816803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.817289 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.817255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.817425 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.817404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.817553 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.817533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.819209 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.819189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.826028 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.826001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msds7\" (UniqueName: \"kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:13.989849 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:13.989752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:14.263937 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:14.263912    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:02:14.266438 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:02:14.266409    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda905798f_96ee_47d5_b193_1fbdd21e75f7.slice/crio-cd8554e748e6f2afd16d3d40e9788f4325e92812012fffc335108ff90a29d313 WatchSource:0}: Error finding container cd8554e748e6f2afd16d3d40e9788f4325e92812012fffc335108ff90a29d313: Status 404 returned error can't find the container with id cd8554e748e6f2afd16d3d40e9788f4325e92812012fffc335108ff90a29d313
Sep 18 20:02:14.742869 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:14.742838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:14.862284 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:14.862241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerStarted","Data":"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"}
Sep 18 20:02:14.862480 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:14.862328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerStarted","Data":"cd8554e748e6f2afd16d3d40e9788f4325e92812012fffc335108ff90a29d313"}
Sep 18 20:02:15.868125 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:15.868086    2583 generic.go:356] "Generic (PLEG): container finished" podID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerID="f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232" exitCode=0
Sep 18 20:02:15.868617 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:15.868169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerDied","Data":"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"}
Sep 18 20:02:16.875110 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:16.875075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerStarted","Data":"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"}
Sep 18 20:02:16.875519 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:16.875131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerStarted","Data":"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"}
Sep 18 20:02:16.875519 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:16.875267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:17.879667 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:17.879629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:17.880765 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:17.880735    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:18.883666 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:18.883626    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:23.888697 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:23.888658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:02:23.889320 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:23.889294    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:23.913519 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:23.913468    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podStartSLOduration=9.811818893 podStartE2EDuration="10.91345379s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.101634897s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:16.915282615 +0000 UTC m=+2869.290708909" watchObservedRunningTime="2026-09-18 20:02:23.91345379 +0000 UTC m=+2876.288880084"
Sep 18 20:02:24.742478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:24.742437    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:33.889917 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:33.889870    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:34.742717 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:34.742679    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:43.889962 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:43.889918    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:44.743514 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:44.743486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:02:51.827072 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:51.827025    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:02:51.827591 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:51.827293    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" containerID="cri-o://b62f77e52ae858db53c747dde579e71414b17f2ea250d27712e7b37f5e01b450" gracePeriod=30
Sep 18 20:02:51.827591 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:51.827394    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kube-rbac-proxy" containerID="cri-o://2db5ed48ccc2ff5dbb8c80f8543c8c9bbdf21a89201f6bfb0311cdd18b1d0b16" gracePeriod=30
Sep 18 20:02:52.035462 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:52.035421    2583 generic.go:356] "Generic (PLEG): container finished" podID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerID="2db5ed48ccc2ff5dbb8c80f8543c8c9bbdf21a89201f6bfb0311cdd18b1d0b16" exitCode=2
Sep 18 20:02:52.035682 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:52.035647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerDied","Data":"2db5ed48ccc2ff5dbb8c80f8543c8c9bbdf21a89201f6bfb0311cdd18b1d0b16"}
Sep 18 20:02:53.890257 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:53.890215    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:02:54.738252 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:54.738210    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 18 20:02:54.742578 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:54.742537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 18 20:02:55.054115 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.054079    2583 generic.go:356] "Generic (PLEG): container finished" podID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerID="b62f77e52ae858db53c747dde579e71414b17f2ea250d27712e7b37f5e01b450" exitCode=0
Sep 18 20:02:55.054596 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.054154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerDied","Data":"b62f77e52ae858db53c747dde579e71414b17f2ea250d27712e7b37f5e01b450"}
Sep 18 20:02:55.073259 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.073234    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:02:55.168921 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.168872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert\") pod \"f435e44a-acbb-4e83-9fe8-322e7773a41a\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") "
Sep 18 20:02:55.169126 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.168965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location\") pod \"f435e44a-acbb-4e83-9fe8-322e7773a41a\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") "
Sep 18 20:02:55.169126 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169028    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") pod \"f435e44a-acbb-4e83-9fe8-322e7773a41a\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") "
Sep 18 20:02:55.169126 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169051    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config\") pod \"f435e44a-acbb-4e83-9fe8-322e7773a41a\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") "
Sep 18 20:02:55.169126 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169083    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfddc\" (UniqueName: \"kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc\") pod \"f435e44a-acbb-4e83-9fe8-322e7773a41a\" (UID: \"f435e44a-acbb-4e83-9fe8-322e7773a41a\") "
Sep 18 20:02:55.169335 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f435e44a-acbb-4e83-9fe8-322e7773a41a" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:55.169408 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169339    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:55.169461 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.169440    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f435e44a-acbb-4e83-9fe8-322e7773a41a" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:02:55.171242 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.171209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc" (OuterVolumeSpecName: "kube-api-access-bfddc") pod "f435e44a-acbb-4e83-9fe8-322e7773a41a" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a"). InnerVolumeSpecName "kube-api-access-bfddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:02:55.171348 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.171211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f435e44a-acbb-4e83-9fe8-322e7773a41a" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:02:55.179397 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.179356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f435e44a-acbb-4e83-9fe8-322e7773a41a" (UID: "f435e44a-acbb-4e83-9fe8-322e7773a41a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:02:55.269841 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.269750    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f435e44a-acbb-4e83-9fe8-322e7773a41a-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:55.269841 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.269787    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:55.269841 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.269799    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfddc\" (UniqueName: \"kubernetes.io/projected/f435e44a-acbb-4e83-9fe8-322e7773a41a-kube-api-access-bfddc\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:55.269841 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:55.269809    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f435e44a-acbb-4e83-9fe8-322e7773a41a-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:02:56.060510 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.060417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw" event={"ID":"f435e44a-acbb-4e83-9fe8-322e7773a41a","Type":"ContainerDied","Data":"78901ac77c085980d0784510d588c6c2f1d141d5a0d3a3771e968c295de01ede"}
Sep 18 20:02:56.060510 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.060453    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"
Sep 18 20:02:56.060510 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.060496    2583 scope.go:117] "RemoveContainer" containerID="2db5ed48ccc2ff5dbb8c80f8543c8c9bbdf21a89201f6bfb0311cdd18b1d0b16"
Sep 18 20:02:56.069054 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.069032    2583 scope.go:117] "RemoveContainer" containerID="b62f77e52ae858db53c747dde579e71414b17f2ea250d27712e7b37f5e01b450"
Sep 18 20:02:56.077316 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.077289    2583 scope.go:117] "RemoveContainer" containerID="f62bac3ea39c79cb13954ea0ca3aa27372f69e51f8ba248ea6493f07eca21dd5"
Sep 18 20:02:56.082130 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.082101    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:02:56.084654 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:56.084626    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6fd5dfd5ff-7pbcw"]
Sep 18 20:02:57.040829 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.040794    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:02:57.041133 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041119    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container"
Sep 18 20:02:57.041185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041135    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container"
Sep 18 20:02:57.041185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="storage-initializer"
Sep 18 20:02:57.041185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041154    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="storage-initializer"
Sep 18 20:02:57.041185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041160    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kube-rbac-proxy"
Sep 18 20:02:57.041185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041166    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kube-rbac-proxy"
Sep 18 20:02:57.041349 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041223    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kserve-container"
Sep 18 20:02:57.041349 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.041236    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" containerName="kube-rbac-proxy"
Sep 18 20:02:57.044258 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.044240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.047237 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.047214    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:02:57.047584 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.047567    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 18 20:02:57.047691 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.047594    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 18 20:02:57.055210 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.055186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:02:57.186183 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.186138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.186183 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.186181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.186670 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.186204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvqk\" (UniqueName: \"kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.186670 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.186269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.186670 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.186354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287092 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287283 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287283 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287283 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:02:57.287233    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 20:02:57.287283 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvqk\" (UniqueName: \"kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287502 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:02:57.287295    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls podName:a61488c6-ed8c-4e30-8b16-f79591f8a252 nodeName:}" failed. No retries permitted until 2026-09-18 20:02:57.787272761 +0000 UTC m=+2910.162699046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 18 20:02:57.287502 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287592 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287874 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.287922 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.287871    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.297168 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.297098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvqk\" (UniqueName: \"kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.791381 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.791328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.793416 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.793377    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f435e44a-acbb-4e83-9fe8-322e7773a41a" path="/var/lib/kubelet/pods/f435e44a-acbb-4e83-9fe8-322e7773a41a/volumes"
Sep 18 20:02:57.799589 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.799553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-65d45d6447-828vt\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:57.955282 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:57.955237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:02:58.229448 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:58.229418    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:02:58.231008 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:02:58.230978    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61488c6_ed8c_4e30_8b16_f79591f8a252.slice/crio-a948f69a0f280b6efe05371d223b6d3c8d215e261163c476b2a641557af784f6 WatchSource:0}: Error finding container a948f69a0f280b6efe05371d223b6d3c8d215e261163c476b2a641557af784f6: Status 404 returned error can't find the container with id a948f69a0f280b6efe05371d223b6d3c8d215e261163c476b2a641557af784f6
Sep 18 20:02:59.076581 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:59.076543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerStarted","Data":"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"}
Sep 18 20:02:59.076797 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:02:59.076605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerStarted","Data":"a948f69a0f280b6efe05371d223b6d3c8d215e261163c476b2a641557af784f6"}
Sep 18 20:03:03.095677 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:03.095646    2583 generic.go:356] "Generic (PLEG): container finished" podID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerID="69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c" exitCode=0
Sep 18 20:03:03.095677 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:03.095682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerDied","Data":"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"}
Sep 18 20:03:03.890061 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:03.890020    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:04.103205 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:04.103164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerStarted","Data":"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"}
Sep 18 20:03:04.103673 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:04.103240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerStarted","Data":"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"}
Sep 18 20:03:04.127778 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:04.127726    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podStartSLOduration=2.5084578459999998 podStartE2EDuration="7.127707182s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.619249336s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:04.126120973 +0000 UTC m=+2916.501547267" watchObservedRunningTime="2026-09-18 20:03:04.127707182 +0000 UTC m=+2916.503133474"
Sep 18 20:03:09.103963 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:09.103920    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:03:09.104524 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:09.104500    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:03:09.105586 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:09.105548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:09.109248 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:09.109227    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:03:09.126604 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:09.126571    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:10.130467 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:10.130431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:13.890122 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:13.890086    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:20.130650 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:20.130610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:23.890196 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:23.890154    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:30.131071 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:30.131017    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:33.889917 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:33.889878    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:35.787388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:35.787315    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:40.130590 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:40.130530    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:45.791998 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:45.791967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:03:50.131236 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:50.131189    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:03:53.677059 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:53.677015    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:03:53.677639 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:53.677268    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" containerID="cri-o://0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0" gracePeriod=30
Sep 18 20:03:53.677639 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:53.677443    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kube-rbac-proxy" containerID="cri-o://29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261" gracePeriod=30
Sep 18 20:03:53.884708 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:53.884660    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 18 20:03:54.334689 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:54.334655    2583 generic.go:356] "Generic (PLEG): container finished" podID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerID="29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261" exitCode=2
Sep 18 20:03:54.334883 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:54.334730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerDied","Data":"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"}
Sep 18 20:03:55.787377 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:55.787324    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 18 20:03:58.819025 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.818999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:03:58.920608 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920515    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msds7\" (UniqueName: \"kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7\") pod \"a905798f-96ee-47d5-b193-1fbdd21e75f7\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") "
Sep 18 20:03:58.920608 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920584    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location\") pod \"a905798f-96ee-47d5-b193-1fbdd21e75f7\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") "
Sep 18 20:03:58.920608 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls\") pod \"a905798f-96ee-47d5-b193-1fbdd21e75f7\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") "
Sep 18 20:03:58.920837 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config\") pod \"a905798f-96ee-47d5-b193-1fbdd21e75f7\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") "
Sep 18 20:03:58.920837 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920646    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert\") pod \"a905798f-96ee-47d5-b193-1fbdd21e75f7\" (UID: \"a905798f-96ee-47d5-b193-1fbdd21e75f7\") "
Sep 18 20:03:58.920929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a905798f-96ee-47d5-b193-1fbdd21e75f7" (UID: "a905798f-96ee-47d5-b193-1fbdd21e75f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:58.921026 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.920996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a905798f-96ee-47d5-b193-1fbdd21e75f7" (UID: "a905798f-96ee-47d5-b193-1fbdd21e75f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:58.921101 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.921082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a905798f-96ee-47d5-b193-1fbdd21e75f7" (UID: "a905798f-96ee-47d5-b193-1fbdd21e75f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:58.922813 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.922788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7" (OuterVolumeSpecName: "kube-api-access-msds7") pod "a905798f-96ee-47d5-b193-1fbdd21e75f7" (UID: "a905798f-96ee-47d5-b193-1fbdd21e75f7"). InnerVolumeSpecName "kube-api-access-msds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:58.922926 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:58.922884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a905798f-96ee-47d5-b193-1fbdd21e75f7" (UID: "a905798f-96ee-47d5-b193-1fbdd21e75f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:59.022046 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.021999    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msds7\" (UniqueName: \"kubernetes.io/projected/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-api-access-msds7\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.022046 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.022037    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a905798f-96ee-47d5-b193-1fbdd21e75f7-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.022046 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.022047    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a905798f-96ee-47d5-b193-1fbdd21e75f7-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.022046 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.022057    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.022334 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.022066    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a905798f-96ee-47d5-b193-1fbdd21e75f7-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:03:59.358828 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.358797    2583 generic.go:356] "Generic (PLEG): container finished" podID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerID="0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0" exitCode=0
Sep 18 20:03:59.359004 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.358871    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"
Sep 18 20:03:59.359004 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.358890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerDied","Data":"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"}
Sep 18 20:03:59.359004 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.358966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z" event={"ID":"a905798f-96ee-47d5-b193-1fbdd21e75f7","Type":"ContainerDied","Data":"cd8554e748e6f2afd16d3d40e9788f4325e92812012fffc335108ff90a29d313"}
Sep 18 20:03:59.359118 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.359007    2583 scope.go:117] "RemoveContainer" containerID="29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"
Sep 18 20:03:59.368609 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.368589    2583 scope.go:117] "RemoveContainer" containerID="0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"
Sep 18 20:03:59.376171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.376152    2583 scope.go:117] "RemoveContainer" containerID="f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"
Sep 18 20:03:59.383151 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.383120    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:03:59.384777 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.384754    2583 scope.go:117] "RemoveContainer" containerID="29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"
Sep 18 20:03:59.385139 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:03:59.385116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261\": container with ID starting with 29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261 not found: ID does not exist" containerID="29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"
Sep 18 20:03:59.385234 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385152    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261"} err="failed to get container status \"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261\": rpc error: code = NotFound desc = could not find container \"29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261\": container with ID starting with 29685f976449ff046ec84e2da8752b065f370075e14e8ccbcc49f2cc7c0ae261 not found: ID does not exist"
Sep 18 20:03:59.385234 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385176    2583 scope.go:117] "RemoveContainer" containerID="0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"
Sep 18 20:03:59.385464 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385447    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-66586f55f9-r6f2z"]
Sep 18 20:03:59.385543 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:03:59.385466    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0\": container with ID starting with 0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0 not found: ID does not exist" containerID="0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"
Sep 18 20:03:59.385543 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385497    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0"} err="failed to get container status \"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0\": rpc error: code = NotFound desc = could not find container \"0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0\": container with ID starting with 0ba402cf5b3c4a0f1c78cefaa41d2db33011b0c7475a3cefbabe4f465f3e99f0 not found: ID does not exist"
Sep 18 20:03:59.385543 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385521    2583 scope.go:117] "RemoveContainer" containerID="f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"
Sep 18 20:03:59.385829 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:03:59.385813    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232\": container with ID starting with f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232 not found: ID does not exist" containerID="f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"
Sep 18 20:03:59.385870 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.385834    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232"} err="failed to get container status \"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232\": rpc error: code = NotFound desc = could not find container \"f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232\": container with ID starting with f44865f66db5cd9856d17e6d5cb06f2aca963c015dbcf6efb0cd72ed69c68232 not found: ID does not exist"
Sep 18 20:03:59.791997 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:03:59.791965    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" path="/var/lib/kubelet/pods/a905798f-96ee-47d5-b193-1fbdd21e75f7/volumes"
Sep 18 20:04:00.131581 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:00.131499    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:04:05.918550 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918518    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918841    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="storage-initializer"
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918853    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="storage-initializer"
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918865    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container"
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918870    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container"
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918884    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kube-rbac-proxy"
Sep 18 20:04:05.918929 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918889    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kube-rbac-proxy"
Sep 18 20:04:05.919140 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918944    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kube-rbac-proxy"
Sep 18 20:04:05.919140 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.918955    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a905798f-96ee-47d5-b193-1fbdd21e75f7" containerName="kserve-container"
Sep 18 20:04:05.923830 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.923810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:05.926784 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.926758    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 18 20:04:05.926970 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.926945    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 18 20:04:05.927077 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.926979    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 18 20:04:05.933208 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.933183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:04:05.978197 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.978139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:05.978197 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.978208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:05.978478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.978244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:05.978478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.978292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:05.978478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:05.978318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75v2m\" (UniqueName: \"kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.078954 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.078918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079114 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.078975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079114 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079114 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079114 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75v2m\" (UniqueName: \"kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079337 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:04:06.079164    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 18 20:04:06.079337 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:04:06.079266    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls podName:8abd237d-6977-4e91-8bb8-3d00885edc70 nodeName:}" failed. No retries permitted until 2026-09-18 20:04:06.579228312 +0000 UTC m=+2978.954654604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 18 20:04:06.079491 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079718 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.079808 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.079786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.089213 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.089187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75v2m\" (UniqueName: \"kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.583770 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.583728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.586271 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.586245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:06.834839 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:06.834732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:07.122667 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:07.122572    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:04:07.125762 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:04:07.125726    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd237d_6977_4e91_8bb8_3d00885edc70.slice/crio-b6dc4b4521f8c6f000e9b95cde4a99ef17c258f96501f99b4a815442ec00b8af WatchSource:0}: Error finding container b6dc4b4521f8c6f000e9b95cde4a99ef17c258f96501f99b4a815442ec00b8af: Status 404 returned error can't find the container with id b6dc4b4521f8c6f000e9b95cde4a99ef17c258f96501f99b4a815442ec00b8af
Sep 18 20:04:07.396689 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:07.396595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerStarted","Data":"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"}
Sep 18 20:04:07.396689 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:07.396656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerStarted","Data":"b6dc4b4521f8c6f000e9b95cde4a99ef17c258f96501f99b4a815442ec00b8af"}
Sep 18 20:04:08.403422 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:08.403383    2583 generic.go:356] "Generic (PLEG): container finished" podID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerID="6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a" exitCode=0
Sep 18 20:04:08.403879 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:08.403428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerDied","Data":"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"}
Sep 18 20:04:08.767249 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:08.767213    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:04:08.767497 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:08.767471    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" containerID="cri-o://bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874" gracePeriod=30
Sep 18 20:04:08.767584 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:08.767515    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kube-rbac-proxy" containerID="cri-o://b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780" gracePeriod=30
Sep 18 20:04:09.109655 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.109554    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 18 20:04:09.417012 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.416897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerStarted","Data":"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"}
Sep 18 20:04:09.417504 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.417131    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:09.417504 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.417202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:09.417504 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.417234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerStarted","Data":"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"}
Sep 18 20:04:09.418342 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.418308    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:09.419302 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.419285    2583 generic.go:356] "Generic (PLEG): container finished" podID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerID="b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780" exitCode=2
Sep 18 20:04:09.419384 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.419344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerDied","Data":"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"}
Sep 18 20:04:09.440478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:09.440426    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podStartSLOduration=3.369100332 podStartE2EDuration="4.44041136s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.071311028s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:09.438226796 +0000 UTC m=+2981.813653088" watchObservedRunningTime="2026-09-18 20:04:09.44041136 +0000 UTC m=+2981.815837652"
Sep 18 20:04:10.130586 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:10.130546    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 18 20:04:10.424969 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:10.424873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:12.211064 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.211036    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:04:12.331013 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.330919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxvqk\" (UniqueName: \"kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk\") pod \"a61488c6-ed8c-4e30-8b16-f79591f8a252\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") "
Sep 18 20:04:12.331013 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.330970    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location\") pod \"a61488c6-ed8c-4e30-8b16-f79591f8a252\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") "
Sep 18 20:04:12.331013 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.330990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert\") pod \"a61488c6-ed8c-4e30-8b16-f79591f8a252\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") "
Sep 18 20:04:12.331422 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.331020    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config\") pod \"a61488c6-ed8c-4e30-8b16-f79591f8a252\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") "
Sep 18 20:04:12.331422 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.331052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") pod \"a61488c6-ed8c-4e30-8b16-f79591f8a252\" (UID: \"a61488c6-ed8c-4e30-8b16-f79591f8a252\") "
Sep 18 20:04:12.331547 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.331463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a61488c6-ed8c-4e30-8b16-f79591f8a252" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:12.331547 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.331471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a61488c6-ed8c-4e30-8b16-f79591f8a252" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:12.333236 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.333203    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk" (OuterVolumeSpecName: "kube-api-access-xxvqk") pod "a61488c6-ed8c-4e30-8b16-f79591f8a252" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252"). InnerVolumeSpecName "kube-api-access-xxvqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:12.333345 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.333295    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a61488c6-ed8c-4e30-8b16-f79591f8a252" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:12.341206 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.341176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a61488c6-ed8c-4e30-8b16-f79591f8a252" (UID: "a61488c6-ed8c-4e30-8b16-f79591f8a252"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:12.431649 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.431610    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61488c6-ed8c-4e30-8b16-f79591f8a252-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:04:12.431649 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.431639    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvqk\" (UniqueName: \"kubernetes.io/projected/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-api-access-xxvqk\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:04:12.431649 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.431652    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a61488c6-ed8c-4e30-8b16-f79591f8a252-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:04:12.431891 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.431661    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:04:12.431891 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.431670    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61488c6-ed8c-4e30-8b16-f79591f8a252-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:04:12.435819 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.435784    2583 generic.go:356] "Generic (PLEG): container finished" podID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerID="bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874" exitCode=0
Sep 18 20:04:12.435945 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.435851    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"
Sep 18 20:04:12.435945 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.435864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerDied","Data":"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"}
Sep 18 20:04:12.435945 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.435930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt" event={"ID":"a61488c6-ed8c-4e30-8b16-f79591f8a252","Type":"ContainerDied","Data":"a948f69a0f280b6efe05371d223b6d3c8d215e261163c476b2a641557af784f6"}
Sep 18 20:04:12.436074 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.435961    2583 scope.go:117] "RemoveContainer" containerID="b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"
Sep 18 20:04:12.444834 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.444811    2583 scope.go:117] "RemoveContainer" containerID="bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"
Sep 18 20:04:12.452935 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.452908    2583 scope.go:117] "RemoveContainer" containerID="69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"
Sep 18 20:04:12.461066 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.461024    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:04:12.464212 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.464187    2583 scope.go:117] "RemoveContainer" containerID="b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"
Sep 18 20:04:12.464631 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:04:12.464599    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780\": container with ID starting with b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780 not found: ID does not exist" containerID="b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"
Sep 18 20:04:12.464781 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.464639    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780"} err="failed to get container status \"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780\": rpc error: code = NotFound desc = could not find container \"b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780\": container with ID starting with b680e9a8078e6baf00e0451cf3fb308d74cb855c256f6d8fe9346291f935a780 not found: ID does not exist"
Sep 18 20:04:12.464781 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.464657    2583 scope.go:117] "RemoveContainer" containerID="bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"
Sep 18 20:04:12.465159 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:04:12.465027    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874\": container with ID starting with bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874 not found: ID does not exist" containerID="bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"
Sep 18 20:04:12.465159 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.465079    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874"} err="failed to get container status \"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874\": rpc error: code = NotFound desc = could not find container \"bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874\": container with ID starting with bc07df538fee796ee5c18097213cb2dbf12e3387069620be3dbfcb94a1562874 not found: ID does not exist"
Sep 18 20:04:12.465159 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.465103    2583 scope.go:117] "RemoveContainer" containerID="69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"
Sep 18 20:04:12.465616 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:04:12.465471    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c\": container with ID starting with 69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c not found: ID does not exist" containerID="69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"
Sep 18 20:04:12.465616 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.465503    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c"} err="failed to get container status \"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c\": rpc error: code = NotFound desc = could not find container \"69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c\": container with ID starting with 69abecb84a805cc2b66cf56071f443e554e23ba7115b19e365a6ba8ff6dc198c not found: ID does not exist"
Sep 18 20:04:12.467248 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:12.467222    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-65d45d6447-828vt"]
Sep 18 20:04:13.791691 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:13.791653    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" path="/var/lib/kubelet/pods/a61488c6-ed8c-4e30-8b16-f79591f8a252/volumes"
Sep 18 20:04:14.004994 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.004949    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:04:14.005312 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005296    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="storage-initializer"
Sep 18 20:04:14.005312 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005311    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="storage-initializer"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005323    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005329    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005335    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kube-rbac-proxy"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kube-rbac-proxy"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005431    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kserve-container"
Sep 18 20:04:14.005493 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.005444    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61488c6-ed8c-4e30-8b16-f79591f8a252" containerName="kube-rbac-proxy"
Sep 18 20:04:14.010164 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.010141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.014745 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.014720    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 18 20:04:14.015044 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.015023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 18 20:04:14.016403 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.016383    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:04:14.024066 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.024036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:04:14.047268 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.047175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.047268 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.047211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.047268 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.047233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.047574 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.047277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.047574 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.047355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrn2\" (UniqueName: \"kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.148716 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.148675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.148716 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.148716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.148993 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.148742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.148993 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.148772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.148993 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.148799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrn2\" (UniqueName: \"kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.149164 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.149143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.149495 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.149476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.149565 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.149512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.151462 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.151438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.176682 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.176653    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrn2\" (UniqueName: \"kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2\") pod \"isvc-pmml-predictor-57685d4d84-bx4tr\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.321134 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.321030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:14.608437 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:14.604181    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:04:14.609524 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:04:14.609491    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a96e68_d08b_4c72_9fff_b00d6771b59c.slice/crio-baa5886294d940f6130da5a6542c2c415be21d93a2fcf295cf0bcb1c0fbaa2d2 WatchSource:0}: Error finding container baa5886294d940f6130da5a6542c2c415be21d93a2fcf295cf0bcb1c0fbaa2d2: Status 404 returned error can't find the container with id baa5886294d940f6130da5a6542c2c415be21d93a2fcf295cf0bcb1c0fbaa2d2
Sep 18 20:04:15.430253 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:15.430224    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:04:15.430920 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:15.430890    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:15.450410 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:15.450378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerStarted","Data":"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"}
Sep 18 20:04:15.450564 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:15.450445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerStarted","Data":"baa5886294d940f6130da5a6542c2c415be21d93a2fcf295cf0bcb1c0fbaa2d2"}
Sep 18 20:04:19.470941 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:19.470908    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerID="3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278" exitCode=0
Sep 18 20:04:19.471340 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:19.470980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerDied","Data":"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"}
Sep 18 20:04:25.431488 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:25.431440    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:27.519422 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:27.519387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerStarted","Data":"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"}
Sep 18 20:04:27.519881 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:27.519451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerStarted","Data":"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"}
Sep 18 20:04:27.519881 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:27.519713    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:27.541209 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:27.541155    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podStartSLOduration=3.040117386 podStartE2EDuration="14.541139033s" totalImagesPullingTime="7.617506304s" totalInitContainerRuntime="3.883515343s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:27.539536826 +0000 UTC m=+2999.914963118" watchObservedRunningTime="2026-09-18 20:04:27.541139033 +0000 UTC m=+2999.916565325"
Sep 18 20:04:28.524603 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:28.524564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:28.525899 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:28.525868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:04:29.529786 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:29.529745    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:04:34.534315 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:34.534282    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:04:34.534928 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:34.534899    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:04:35.431593 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:35.431556    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:44.534880 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:44.534840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:04:45.430966 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:45.430927    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:04:54.535866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:54.535815    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:04:55.431599 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:04:55.431559    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:05:04.535440 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:04.535397    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:05:05.431291 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:05.431254    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:05:14.535413 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:14.535299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:05:15.430840 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:15.430804    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:05:24.535650 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:24.535609    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:05:25.431729 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:25.431684    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 18 20:05:31.893808 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:31.893771    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:05:31.896873 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:31.896851    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:05:33.791983 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:33.791954    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:05:34.535460 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:34.535417    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:05:35.918434 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:35.918397    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:05:35.918959 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:35.918670    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container" containerID="cri-o://008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26" gracePeriod=30
Sep 18 20:05:35.918959 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:35.918712    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kube-rbac-proxy" containerID="cri-o://22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52" gracePeriod=30
Sep 18 20:05:36.835714 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:36.835677    2583 generic.go:356] "Generic (PLEG): container finished" podID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerID="22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52" exitCode=2
Sep 18 20:05:36.835911 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:36.835754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerDied","Data":"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"}
Sep 18 20:05:40.425626 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:40.425579    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 18 20:05:41.461911 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.461886    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:05:41.508644 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.508611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert\") pod \"8abd237d-6977-4e91-8bb8-3d00885edc70\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") "
Sep 18 20:05:41.508816 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.508662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config\") pod \"8abd237d-6977-4e91-8bb8-3d00885edc70\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") "
Sep 18 20:05:41.508816 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.508739    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") pod \"8abd237d-6977-4e91-8bb8-3d00885edc70\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") "
Sep 18 20:05:41.508816 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.508796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location\") pod \"8abd237d-6977-4e91-8bb8-3d00885edc70\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") "
Sep 18 20:05:41.509000 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.508830    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75v2m\" (UniqueName: \"kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m\") pod \"8abd237d-6977-4e91-8bb8-3d00885edc70\" (UID: \"8abd237d-6977-4e91-8bb8-3d00885edc70\") "
Sep 18 20:05:41.509101 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.509065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8abd237d-6977-4e91-8bb8-3d00885edc70" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:41.509240 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.509126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8abd237d-6977-4e91-8bb8-3d00885edc70" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:05:41.509240 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.509160    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8abd237d-6977-4e91-8bb8-3d00885edc70" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:05:41.510845 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.510820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8abd237d-6977-4e91-8bb8-3d00885edc70" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:05:41.510962 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.510942    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m" (OuterVolumeSpecName: "kube-api-access-75v2m") pod "8abd237d-6977-4e91-8bb8-3d00885edc70" (UID: "8abd237d-6977-4e91-8bb8-3d00885edc70"). InnerVolumeSpecName "kube-api-access-75v2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:05:41.609530 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.609443    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8abd237d-6977-4e91-8bb8-3d00885edc70-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:05:41.609530 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.609474    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8abd237d-6977-4e91-8bb8-3d00885edc70-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:05:41.609530 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.609485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75v2m\" (UniqueName: \"kubernetes.io/projected/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-api-access-75v2m\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:05:41.609530 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.609495    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:05:41.609530 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.609504    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8abd237d-6977-4e91-8bb8-3d00885edc70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:05:41.859302 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.859272    2583 generic.go:356] "Generic (PLEG): container finished" podID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerID="008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26" exitCode=0
Sep 18 20:05:41.859492 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.859333    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"
Sep 18 20:05:41.859492 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.859345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerDied","Data":"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"}
Sep 18 20:05:41.859492 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.859428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2" event={"ID":"8abd237d-6977-4e91-8bb8-3d00885edc70","Type":"ContainerDied","Data":"b6dc4b4521f8c6f000e9b95cde4a99ef17c258f96501f99b4a815442ec00b8af"}
Sep 18 20:05:41.859492 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.859455    2583 scope.go:117] "RemoveContainer" containerID="22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"
Sep 18 20:05:41.867800 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.867783    2583 scope.go:117] "RemoveContainer" containerID="008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"
Sep 18 20:05:41.875308 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.875287    2583 scope.go:117] "RemoveContainer" containerID="6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"
Sep 18 20:05:41.878324 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.878298    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:05:41.882246 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.882225    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5d7666c88d-hrxv2"]
Sep 18 20:05:41.883283 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.883268    2583 scope.go:117] "RemoveContainer" containerID="22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"
Sep 18 20:05:41.883579 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:05:41.883557    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52\": container with ID starting with 22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52 not found: ID does not exist" containerID="22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"
Sep 18 20:05:41.883641 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.883587    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52"} err="failed to get container status \"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52\": rpc error: code = NotFound desc = could not find container \"22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52\": container with ID starting with 22d2155706fc2e5b67032cfb5ff76d5838574903e0f064b2aaa00265db8baa52 not found: ID does not exist"
Sep 18 20:05:41.883641 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.883608    2583 scope.go:117] "RemoveContainer" containerID="008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"
Sep 18 20:05:41.883890 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:05:41.883874    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26\": container with ID starting with 008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26 not found: ID does not exist" containerID="008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"
Sep 18 20:05:41.883932 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.883895    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26"} err="failed to get container status \"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26\": rpc error: code = NotFound desc = could not find container \"008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26\": container with ID starting with 008776096ce70af158262249b96d818c2fd3ae1963317701f050433d2d5f0a26 not found: ID does not exist"
Sep 18 20:05:41.883932 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.883912    2583 scope.go:117] "RemoveContainer" containerID="6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"
Sep 18 20:05:41.884139 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:05:41.884122    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a\": container with ID starting with 6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a not found: ID does not exist" containerID="6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"
Sep 18 20:05:41.884183 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:41.884145    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a"} err="failed to get container status \"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a\": rpc error: code = NotFound desc = could not find container \"6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a\": container with ID starting with 6451baa73e9085d0913c9e199c3ae6ec4a410f9be5c0a5fbef4f252b06dc098a not found: ID does not exist"
Sep 18 20:05:43.791219 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:43.791178    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" path="/var/lib/kubelet/pods/8abd237d-6977-4e91-8bb8-3d00885edc70/volumes"
Sep 18 20:05:44.534762 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:44.534721    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:05:54.535630 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:05:54.535584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 18 20:06:00.788031 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:00.788000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:06:05.933864 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:05.933821    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:06:05.934282 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:05.934148    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container" containerID="cri-o://079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e" gracePeriod=30
Sep 18 20:06:05.934282 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:05.934231    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kube-rbac-proxy" containerID="cri-o://815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b" gracePeriod=30
Sep 18 20:06:06.967313 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:06.967275    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerID="815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b" exitCode=2
Sep 18 20:06:06.967705 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:06.967329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerDied","Data":"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"}
Sep 18 20:06:09.530611 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:09.530564    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 18 20:06:10.384286 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.384260    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:06:10.446768 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.446723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztrn2\" (UniqueName: \"kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2\") pod \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") "
Sep 18 20:06:10.446949 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.446780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config\") pod \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") "
Sep 18 20:06:10.446949 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.446834    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location\") pod \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") "
Sep 18 20:06:10.446949 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.446866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls\") pod \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") "
Sep 18 20:06:10.446949 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.446907    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert\") pod \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\" (UID: \"e2a96e68-d08b-4c72-9fff-b00d6771b59c\") "
Sep 18 20:06:10.447199 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.447166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2a96e68-d08b-4c72-9fff-b00d6771b59c" (UID: "e2a96e68-d08b-4c72-9fff-b00d6771b59c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:10.447332 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.447279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2a96e68-d08b-4c72-9fff-b00d6771b59c" (UID: "e2a96e68-d08b-4c72-9fff-b00d6771b59c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:10.447420 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.447342    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2a96e68-d08b-4c72-9fff-b00d6771b59c" (UID: "e2a96e68-d08b-4c72-9fff-b00d6771b59c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:10.449083 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.449023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2" (OuterVolumeSpecName: "kube-api-access-ztrn2") pod "e2a96e68-d08b-4c72-9fff-b00d6771b59c" (UID: "e2a96e68-d08b-4c72-9fff-b00d6771b59c"). InnerVolumeSpecName "kube-api-access-ztrn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:10.449083 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.449023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2a96e68-d08b-4c72-9fff-b00d6771b59c" (UID: "e2a96e68-d08b-4c72-9fff-b00d6771b59c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:10.547606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.547567    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:06:10.547606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.547596    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2a96e68-d08b-4c72-9fff-b00d6771b59c-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:06:10.547606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.547605    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:06:10.547606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.547613    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrn2\" (UniqueName: \"kubernetes.io/projected/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-api-access-ztrn2\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:06:10.548062 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.547622    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2a96e68-d08b-4c72-9fff-b00d6771b59c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:06:10.986071 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.986035    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerID="079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e" exitCode=0
Sep 18 20:06:10.986251 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.986089    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"
Sep 18 20:06:10.986251 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.986112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerDied","Data":"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"}
Sep 18 20:06:10.986251 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.986184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr" event={"ID":"e2a96e68-d08b-4c72-9fff-b00d6771b59c","Type":"ContainerDied","Data":"baa5886294d940f6130da5a6542c2c415be21d93a2fcf295cf0bcb1c0fbaa2d2"}
Sep 18 20:06:10.986251 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.986212    2583 scope.go:117] "RemoveContainer" containerID="815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"
Sep 18 20:06:10.995573 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:10.995548    2583 scope.go:117] "RemoveContainer" containerID="079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"
Sep 18 20:06:11.003685 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.003665    2583 scope.go:117] "RemoveContainer" containerID="3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"
Sep 18 20:06:11.011472 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.011453    2583 scope.go:117] "RemoveContainer" containerID="815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"
Sep 18 20:06:11.011781 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:06:11.011762    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b\": container with ID starting with 815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b not found: ID does not exist" containerID="815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"
Sep 18 20:06:11.011836 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.011791    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b"} err="failed to get container status \"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b\": rpc error: code = NotFound desc = could not find container \"815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b\": container with ID starting with 815709e3b469e1f5a23dddbcec1914e91f484e322891df166f501c9b528e945b not found: ID does not exist"
Sep 18 20:06:11.011836 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.011814    2583 scope.go:117] "RemoveContainer" containerID="079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"
Sep 18 20:06:11.012085 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:06:11.012063    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e\": container with ID starting with 079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e not found: ID does not exist" containerID="079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"
Sep 18 20:06:11.012119 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.012093    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e"} err="failed to get container status \"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e\": rpc error: code = NotFound desc = could not find container \"079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e\": container with ID starting with 079464bb0b4a8378eb389b8ad825d10a782d446975562eb1cd741411f763fc6e not found: ID does not exist"
Sep 18 20:06:11.012119 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.012110    2583 scope.go:117] "RemoveContainer" containerID="3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"
Sep 18 20:06:11.012355 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:06:11.012336    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278\": container with ID starting with 3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278 not found: ID does not exist" containerID="3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"
Sep 18 20:06:11.012424 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.012376    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278"} err="failed to get container status \"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278\": rpc error: code = NotFound desc = could not find container \"3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278\": container with ID starting with 3e546f437b79405a92bf05af75815f39666acf2a2ca91c5b47ed4cfdf96be278 not found: ID does not exist"
Sep 18 20:06:11.022322 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.022287    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:06:11.028983 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.028954    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-57685d4d84-bx4tr"]
Sep 18 20:06:11.147813 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.147778    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:06:11.148091 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148080    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148094    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148103    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148108    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148121    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148127    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148134    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148148 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148139    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="storage-initializer"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="storage-initializer"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148165    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="storage-initializer"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="storage-initializer"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148220    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148227    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kube-rbac-proxy"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148235    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8abd237d-6977-4e91-8bb8-3d00885edc70" containerName="kserve-container"
Sep 18 20:06:11.148388 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.148242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" containerName="kserve-container"
Sep 18 20:06:11.151831 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.151811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.154828 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.154802    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 18 20:06:11.154964 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.154908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 18 20:06:11.154964 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.154928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:06:11.154964 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.154947    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:06:11.155127 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.155015    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:06:11.155190 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.155160    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:06:11.155811 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.155796    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5b562\""
Sep 18 20:06:11.168219 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.168193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:06:11.253243 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.253210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.253428 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.253267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.253428 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.253336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whg7x\" (UniqueName: \"kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.253428 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.253401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.253535 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.253441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.354750 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.354712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.354750 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.354751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.354966 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.354875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.354966 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.354951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whg7x\" (UniqueName: \"kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.355056 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.354994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.355354 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.355330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.355550 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.355528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.355614 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.355568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.357252 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.357230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.364284 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.364263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whg7x\" (UniqueName: \"kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x\") pod \"isvc-pmml-runtime-predictor-58cc747bcd-bfdcd\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.462228 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.462185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:11.754165 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.754128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:06:11.756309 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:06:11.756281    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde78ba44_3d3b_4075_b711_9d3665afaba3.slice/crio-ca1c500d494c93d5236460635d974ce6a28e2a51c5ea44a67c4f3273994b4247 WatchSource:0}: Error finding container ca1c500d494c93d5236460635d974ce6a28e2a51c5ea44a67c4f3273994b4247: Status 404 returned error can't find the container with id ca1c500d494c93d5236460635d974ce6a28e2a51c5ea44a67c4f3273994b4247
Sep 18 20:06:11.791964 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.791916    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a96e68-d08b-4c72-9fff-b00d6771b59c" path="/var/lib/kubelet/pods/e2a96e68-d08b-4c72-9fff-b00d6771b59c/volumes"
Sep 18 20:06:11.992569 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.992528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerStarted","Data":"bdde529ff8af7c746106090539b943911ea667f6350c3c6cdb0dfefb98212bec"}
Sep 18 20:06:11.992880 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:11.992606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerStarted","Data":"ca1c500d494c93d5236460635d974ce6a28e2a51c5ea44a67c4f3273994b4247"}
Sep 18 20:06:16.011815 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:16.011773    2583 generic.go:356] "Generic (PLEG): container finished" podID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerID="bdde529ff8af7c746106090539b943911ea667f6350c3c6cdb0dfefb98212bec" exitCode=0
Sep 18 20:06:16.012264 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:16.011845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerDied","Data":"bdde529ff8af7c746106090539b943911ea667f6350c3c6cdb0dfefb98212bec"}
Sep 18 20:06:17.018202 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.018167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerStarted","Data":"27a4372760c7befef90fbdaf10063fe0c377dc30ad30b33ee77f10792e8ed309"}
Sep 18 20:06:17.018606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.018229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerStarted","Data":"94b13c6d9c313e1d33a0e377305d5d3156d24e22828cceecb30e0634b3f643aa"}
Sep 18 20:06:17.018606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.018507    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:17.018702 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.018647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:17.019849 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.019820    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:06:17.065229 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:17.065172    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podStartSLOduration=2.194875717 podStartE2EDuration="6.065156565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870280848s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:06:17.06476099 +0000 UTC m=+3109.440187283" watchObservedRunningTime="2026-09-18 20:06:17.065156565 +0000 UTC m=+3109.440582859"
Sep 18 20:06:18.022471 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:18.022432    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:06:23.027427 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:23.027396    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:06:23.027918 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:23.027888    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:06:33.028020 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:33.027971    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:06:43.028279 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:43.028198    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:06:53.028116 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:06:53.028067    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:03.027952 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:03.027905    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:13.028833 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:13.028796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:23.028812 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:23.028769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:33.028822 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:33.028777    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:36.787055 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:36.787014    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:46.788513 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:46.788478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:07:53.018999 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.018952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:07:53.019653 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.019180    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" containerID="cri-o://94b13c6d9c313e1d33a0e377305d5d3156d24e22828cceecb30e0634b3f643aa" gracePeriod=30
Sep 18 20:07:53.019653 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.019241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kube-rbac-proxy" containerID="cri-o://27a4372760c7befef90fbdaf10063fe0c377dc30ad30b33ee77f10792e8ed309" gracePeriod=30
Sep 18 20:07:53.023687 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.023651    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": read tcp 10.134.0.2:46824->10.134.0.64:8643: read: connection reset by peer"
Sep 18 20:07:53.422293 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.422205    2583 generic.go:356] "Generic (PLEG): container finished" podID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerID="27a4372760c7befef90fbdaf10063fe0c377dc30ad30b33ee77f10792e8ed309" exitCode=2
Sep 18 20:07:53.422293 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:53.422248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerDied","Data":"27a4372760c7befef90fbdaf10063fe0c377dc30ad30b33ee77f10792e8ed309"}
Sep 18 20:07:56.787508 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:56.787459    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 18 20:07:57.440079 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.440044    2583 generic.go:356] "Generic (PLEG): container finished" podID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerID="94b13c6d9c313e1d33a0e377305d5d3156d24e22828cceecb30e0634b3f643aa" exitCode=0
Sep 18 20:07:57.440254 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.440098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerDied","Data":"94b13c6d9c313e1d33a0e377305d5d3156d24e22828cceecb30e0634b3f643aa"}
Sep 18 20:07:57.465404 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.465374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:07:57.552771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.552666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert\") pod \"de78ba44-3d3b-4075-b711-9d3665afaba3\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") "
Sep 18 20:07:57.552771 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.552744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location\") pod \"de78ba44-3d3b-4075-b711-9d3665afaba3\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") "
Sep 18 20:07:57.553014 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.552803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whg7x\" (UniqueName: \"kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x\") pod \"de78ba44-3d3b-4075-b711-9d3665afaba3\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") "
Sep 18 20:07:57.553014 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.552828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls\") pod \"de78ba44-3d3b-4075-b711-9d3665afaba3\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") "
Sep 18 20:07:57.553014 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.552863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config\") pod \"de78ba44-3d3b-4075-b711-9d3665afaba3\" (UID: \"de78ba44-3d3b-4075-b711-9d3665afaba3\") "
Sep 18 20:07:57.553153 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.553102    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de78ba44-3d3b-4075-b711-9d3665afaba3" (UID: "de78ba44-3d3b-4075-b711-9d3665afaba3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:57.553153 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.553110    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de78ba44-3d3b-4075-b711-9d3665afaba3" (UID: "de78ba44-3d3b-4075-b711-9d3665afaba3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:57.553341 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.553311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de78ba44-3d3b-4075-b711-9d3665afaba3" (UID: "de78ba44-3d3b-4075-b711-9d3665afaba3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:57.554990 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.554963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de78ba44-3d3b-4075-b711-9d3665afaba3" (UID: "de78ba44-3d3b-4075-b711-9d3665afaba3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:57.555088 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.555037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x" (OuterVolumeSpecName: "kube-api-access-whg7x") pod "de78ba44-3d3b-4075-b711-9d3665afaba3" (UID: "de78ba44-3d3b-4075-b711-9d3665afaba3"). InnerVolumeSpecName "kube-api-access-whg7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:57.653470 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.653434    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whg7x\" (UniqueName: \"kubernetes.io/projected/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-api-access-whg7x\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:07:57.653470 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.653464    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de78ba44-3d3b-4075-b711-9d3665afaba3-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:07:57.653470 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.653473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:07:57.653470 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.653482    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de78ba44-3d3b-4075-b711-9d3665afaba3-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:07:57.653746 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:57.653492    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de78ba44-3d3b-4075-b711-9d3665afaba3-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:07:58.231976 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.231937    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232250    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="storage-initializer"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232261    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="storage-initializer"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232277    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kube-rbac-proxy"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232283    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kube-rbac-proxy"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232290    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container"
Sep 18 20:07:58.232355 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232350    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kserve-container"
Sep 18 20:07:58.232632 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.232376    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" containerName="kube-rbac-proxy"
Sep 18 20:07:58.234602 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.234584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.238131 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.238104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cf00b2-predictor-serving-cert\""
Sep 18 20:07:58.241101 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.241077    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cf00b2-kube-rbac-proxy-sar-config\""
Sep 18 20:07:58.255032 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.254999    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:07:58.359348 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.359305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.359348 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.359351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cn9\" (UniqueName: \"kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.359606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.359396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.359606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.359463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.359606 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.359556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.446248 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.446213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd" event={"ID":"de78ba44-3d3b-4075-b711-9d3665afaba3","Type":"ContainerDied","Data":"ca1c500d494c93d5236460635d974ce6a28e2a51c5ea44a67c4f3273994b4247"}
Sep 18 20:07:58.446248 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.446236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"
Sep 18 20:07:58.446643 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.446294    2583 scope.go:117] "RemoveContainer" containerID="27a4372760c7befef90fbdaf10063fe0c377dc30ad30b33ee77f10792e8ed309"
Sep 18 20:07:58.454852 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.454834    2583 scope.go:117] "RemoveContainer" containerID="94b13c6d9c313e1d33a0e377305d5d3156d24e22828cceecb30e0634b3f643aa"
Sep 18 20:07:58.460094 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cn9\" (UniqueName: \"kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460185 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460339 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460663 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460900 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.460982 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.460876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.462643 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.462619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.462805 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.462788    2583 scope.go:117] "RemoveContainer" containerID="bdde529ff8af7c746106090539b943911ea667f6350c3c6cdb0dfefb98212bec"
Sep 18 20:07:58.466586 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.466560    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:07:58.469344 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.469321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cn9\" (UniqueName: \"kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9\") pod \"isvc-primary-cf00b2-predictor-7d877cbff-pdv62\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") " pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.469914 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.469889    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-58cc747bcd-bfdcd"]
Sep 18 20:07:58.544274 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.544233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:07:58.819174 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:58.819145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:07:58.821819 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:07:58.821792    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759717e8_7b06_4ad1_bfa9_84eb251430fc.slice/crio-d8b37a98c9757b1a540b1c46dce04472960a716665dfcba2451ed9f954b5b392 WatchSource:0}: Error finding container d8b37a98c9757b1a540b1c46dce04472960a716665dfcba2451ed9f954b5b392: Status 404 returned error can't find the container with id d8b37a98c9757b1a540b1c46dce04472960a716665dfcba2451ed9f954b5b392
Sep 18 20:07:59.453836 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:59.453800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerStarted","Data":"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"}
Sep 18 20:07:59.454330 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:59.453870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerStarted","Data":"d8b37a98c9757b1a540b1c46dce04472960a716665dfcba2451ed9f954b5b392"}
Sep 18 20:07:59.793757 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:07:59.793721    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de78ba44-3d3b-4075-b711-9d3665afaba3" path="/var/lib/kubelet/pods/de78ba44-3d3b-4075-b711-9d3665afaba3/volumes"
Sep 18 20:08:03.472426 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:03.472390    2583 generic.go:356] "Generic (PLEG): container finished" podID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerID="93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444" exitCode=0
Sep 18 20:08:03.472843 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:03.472460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerDied","Data":"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"}
Sep 18 20:08:04.479593 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:04.479559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerStarted","Data":"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"}
Sep 18 20:08:04.480002 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:04.479627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerStarted","Data":"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"}
Sep 18 20:08:04.480002 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:04.479921    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:08:04.480095 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:04.480075    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:08:04.481414 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:04.481387    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:05.483765 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:05.483730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:10.489129 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:10.489048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:08:10.489612 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:10.489576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:10.520428 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:10.520350    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podStartSLOduration=8.913159277 podStartE2EDuration="12.520335204s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.607175927s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:08:04.52364625 +0000 UTC m=+3216.899072542" watchObservedRunningTime="2026-09-18 20:08:10.520335204 +0000 UTC m=+3222.895761496"
Sep 18 20:08:20.490521 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:20.490478    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:30.490066 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:30.490022    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:40.490286 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:40.490246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:08:50.490469 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:08:50.490432    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:09:00.490542 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:00.490504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:09:10.490099 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:10.490058    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:09:14.786919 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:14.786871    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:09:24.788109 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:24.788074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:09:43.463604 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.463512    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:09:43.467134 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.467112    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.470719 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.470687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cf00b2\""
Sep 18 20:09:43.471018 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.470985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cf00b2-dockercfg-7626f\""
Sep 18 20:09:43.471163 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.471128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cf00b2-predictor-serving-cert\""
Sep 18 20:09:43.471277 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.471255    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cf00b2-kube-rbac-proxy-sar-config\""
Sep 18 20:09:43.486332 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.486300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:09:43.551927 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.551894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.552124 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.551951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.552124 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.552044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwfx\" (UniqueName: \"kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.552124 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.552088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.552265 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.552156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.652910 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.652865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653122 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.652928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653122 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.652975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653122 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.653017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653122 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.653076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwfx\" (UniqueName: \"kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653491 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.653468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653690 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.653667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.653752 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.653686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.655483 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.655467    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.668183 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.668155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwfx\" (UniqueName: \"kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx\") pod \"isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") " pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:43.777521 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:43.777474    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:09:44.060858 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:44.060830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:09:44.063665 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:09:44.063634    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6785a832_68c9_4fe0_8fe5_d10cce162ad1.slice/crio-7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1 WatchSource:0}: Error finding container 7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1: Status 404 returned error can't find the container with id 7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1
Sep 18 20:09:44.908603 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:44.908561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerStarted","Data":"d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"}
Sep 18 20:09:44.909030 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:44.908629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerStarted","Data":"7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1"}
Sep 18 20:09:49.931434 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:49.931406    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/0.log"
Sep 18 20:09:49.931856 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:49.931442    2583 generic.go:356] "Generic (PLEG): container finished" podID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerID="d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987" exitCode=1
Sep 18 20:09:49.931856 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:49.931478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerDied","Data":"d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"}
Sep 18 20:09:50.937655 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:50.937628    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/0.log"
Sep 18 20:09:50.938045 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:50.937695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerStarted","Data":"b2f69498af3f9fef3b47e93200cd83ed2ab5791cce25cdc92392111230dbd042"}
Sep 18 20:09:55.959136 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959062    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/1.log"
Sep 18 20:09:55.959548 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959451    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/0.log"
Sep 18 20:09:55.959548 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959479    2583 generic.go:356] "Generic (PLEG): container finished" podID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerID="b2f69498af3f9fef3b47e93200cd83ed2ab5791cce25cdc92392111230dbd042" exitCode=1
Sep 18 20:09:55.959548 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerDied","Data":"b2f69498af3f9fef3b47e93200cd83ed2ab5791cce25cdc92392111230dbd042"}
Sep 18 20:09:55.959660 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959570    2583 scope.go:117] "RemoveContainer" containerID="d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"
Sep 18 20:09:55.959920 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.959901    2583 scope.go:117] "RemoveContainer" containerID="d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"
Sep 18 20:09:55.970973 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:09:55.970949    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_kserve-ci-e2e-test(6785a832-68c9-4fe0-8fe5-d10cce162ad1)\"" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1"
Sep 18 20:09:55.971142 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:09:55.971117    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_kserve-ci-e2e-test_6785a832-68c9-4fe0-8fe5-d10cce162ad1_0 in pod sandbox 7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1 from index: no such id: 'd26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987'" containerID="d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"
Sep 18 20:09:55.971209 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:55.971154    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_kserve-ci-e2e-test_6785a832-68c9-4fe0-8fe5-d10cce162ad1_0 in pod sandbox 7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1 from index: no such id: 'd26bcd0b2901051b4bdf735274783709d94d4a6c39cfe3b181b95cee7e108987'"
Sep 18 20:09:56.965568 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:09:56.965541    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/1.log"
Sep 18 20:10:01.533035 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.533003    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:10:01.560779 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.560737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:10:01.561150 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.561119    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" containerID="cri-o://5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b" gracePeriod=30
Sep 18 20:10:01.561264 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.561207    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kube-rbac-proxy" containerID="cri-o://847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b" gracePeriod=30
Sep 18 20:10:01.682163 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.682141    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/1.log"
Sep 18 20:10:01.684315 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.684299    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:10:01.808490 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808395    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config\") pod \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") "
Sep 18 20:10:01.808490 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808462    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location\") pod \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") "
Sep 18 20:10:01.808490 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls\") pod \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") "
Sep 18 20:10:01.808787 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert\") pod \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") "
Sep 18 20:10:01.808787 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808567    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlwfx\" (UniqueName: \"kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx\") pod \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\" (UID: \"6785a832-68c9-4fe0-8fe5-d10cce162ad1\") "
Sep 18 20:10:01.808787 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6785a832-68c9-4fe0-8fe5-d10cce162ad1" (UID: "6785a832-68c9-4fe0-8fe5-d10cce162ad1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:01.808900 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808839    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6785a832-68c9-4fe0-8fe5-d10cce162ad1" (UID: "6785a832-68c9-4fe0-8fe5-d10cce162ad1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:01.808937 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.808919    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6785a832-68c9-4fe0-8fe5-d10cce162ad1" (UID: "6785a832-68c9-4fe0-8fe5-d10cce162ad1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:01.810702 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.810678    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6785a832-68c9-4fe0-8fe5-d10cce162ad1" (UID: "6785a832-68c9-4fe0-8fe5-d10cce162ad1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:01.810790 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.810704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx" (OuterVolumeSpecName: "kube-api-access-rlwfx") pod "6785a832-68c9-4fe0-8fe5-d10cce162ad1" (UID: "6785a832-68c9-4fe0-8fe5-d10cce162ad1"). InnerVolumeSpecName "kube-api-access-rlwfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:01.910033 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.909993    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6785a832-68c9-4fe0-8fe5-d10cce162ad1-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:01.910033 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.910028    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:01.910033 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.910037    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwfx\" (UniqueName: \"kubernetes.io/projected/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-api-access-rlwfx\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:01.910033 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.910046    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:01.910314 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.910055    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6785a832-68c9-4fe0-8fe5-d10cce162ad1-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:01.988020 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.987991    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr_6785a832-68c9-4fe0-8fe5-d10cce162ad1/storage-initializer/1.log"
Sep 18 20:10:01.988195 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.988097    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"
Sep 18 20:10:01.988195 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.988126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr" event={"ID":"6785a832-68c9-4fe0-8fe5-d10cce162ad1","Type":"ContainerDied","Data":"7c89c51c59d877693196683da605c19256509ea0f9480371c997a002bbfe03f1"}
Sep 18 20:10:01.988295 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.988211    2583 scope.go:117] "RemoveContainer" containerID="b2f69498af3f9fef3b47e93200cd83ed2ab5791cce25cdc92392111230dbd042"
Sep 18 20:10:01.990770 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.990747    2583 generic.go:356] "Generic (PLEG): container finished" podID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerID="847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b" exitCode=2
Sep 18 20:10:01.990902 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:01.990821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerDied","Data":"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"}
Sep 18 20:10:02.034149 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:02.034112    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:10:02.038700 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:02.038667    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cf00b2-predictor-656f4cd6fb-95bsr"]
Sep 18 20:10:03.792506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:03.792476    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" path="/var/lib/kubelet/pods/6785a832-68c9-4fe0-8fe5-d10cce162ad1/volumes"
Sep 18 20:10:04.787081 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:04.787037    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 18 20:10:05.484577 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:05.484536    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 18 20:10:06.920508 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:06.920480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:10:07.014549 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.014514    2583 generic.go:356] "Generic (PLEG): container finished" podID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerID="5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b" exitCode=0
Sep 18 20:10:07.014759 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.014577    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"
Sep 18 20:10:07.014759 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.014598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerDied","Data":"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"}
Sep 18 20:10:07.014759 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.014708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62" event={"ID":"759717e8-7b06-4ad1-bfa9-84eb251430fc","Type":"ContainerDied","Data":"d8b37a98c9757b1a540b1c46dce04472960a716665dfcba2451ed9f954b5b392"}
Sep 18 20:10:07.014759 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.014745    2583 scope.go:117] "RemoveContainer" containerID="847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"
Sep 18 20:10:07.023248 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.023226    2583 scope.go:117] "RemoveContainer" containerID="5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"
Sep 18 20:10:07.031387 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.031346    2583 scope.go:117] "RemoveContainer" containerID="93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"
Sep 18 20:10:07.041561 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.041539    2583 scope.go:117] "RemoveContainer" containerID="847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"
Sep 18 20:10:07.041876 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:07.041857    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b\": container with ID starting with 847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b not found: ID does not exist" containerID="847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"
Sep 18 20:10:07.041924 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.041886    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b"} err="failed to get container status \"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b\": rpc error: code = NotFound desc = could not find container \"847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b\": container with ID starting with 847a88988f14aa6e03305becf8cadea392c21c0acd8e94556eb46cac1922fb6b not found: ID does not exist"
Sep 18 20:10:07.041924 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.041904    2583 scope.go:117] "RemoveContainer" containerID="5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"
Sep 18 20:10:07.042142 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:07.042124    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b\": container with ID starting with 5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b not found: ID does not exist" containerID="5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"
Sep 18 20:10:07.042189 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.042148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b"} err="failed to get container status \"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b\": rpc error: code = NotFound desc = could not find container \"5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b\": container with ID starting with 5638977794ed422750f342e64d1854f8ac8329f3ea7bb70fa345fa00e2c79d4b not found: ID does not exist"
Sep 18 20:10:07.042189 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.042161    2583 scope.go:117] "RemoveContainer" containerID="93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"
Sep 18 20:10:07.042447 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:07.042430    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444\": container with ID starting with 93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444 not found: ID does not exist" containerID="93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"
Sep 18 20:10:07.042508 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.042452    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444"} err="failed to get container status \"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444\": rpc error: code = NotFound desc = could not find container \"93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444\": container with ID starting with 93c7ecd6b5f51fd5abbefec9aaae4fd0be802887ce5cda2d6a138689ee9a3444 not found: ID does not exist"
Sep 18 20:10:07.051767 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.051747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7cn9\" (UniqueName: \"kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9\") pod \"759717e8-7b06-4ad1-bfa9-84eb251430fc\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") "
Sep 18 20:10:07.051898 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.051803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location\") pod \"759717e8-7b06-4ad1-bfa9-84eb251430fc\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") "
Sep 18 20:10:07.051898 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.051861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config\") pod \"759717e8-7b06-4ad1-bfa9-84eb251430fc\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") "
Sep 18 20:10:07.051898 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.051891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls\") pod \"759717e8-7b06-4ad1-bfa9-84eb251430fc\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") "
Sep 18 20:10:07.052068 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.051914    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert\") pod \"759717e8-7b06-4ad1-bfa9-84eb251430fc\" (UID: \"759717e8-7b06-4ad1-bfa9-84eb251430fc\") "
Sep 18 20:10:07.052199 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.052171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "759717e8-7b06-4ad1-bfa9-84eb251430fc" (UID: "759717e8-7b06-4ad1-bfa9-84eb251430fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:07.052268 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.052246    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "759717e8-7b06-4ad1-bfa9-84eb251430fc" (UID: "759717e8-7b06-4ad1-bfa9-84eb251430fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:07.052456 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.052426    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "759717e8-7b06-4ad1-bfa9-84eb251430fc" (UID: "759717e8-7b06-4ad1-bfa9-84eb251430fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:10:07.054007 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.053976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9" (OuterVolumeSpecName: "kube-api-access-k7cn9") pod "759717e8-7b06-4ad1-bfa9-84eb251430fc" (UID: "759717e8-7b06-4ad1-bfa9-84eb251430fc"). InnerVolumeSpecName "kube-api-access-k7cn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:07.054104 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.054080    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "759717e8-7b06-4ad1-bfa9-84eb251430fc" (UID: "759717e8-7b06-4ad1-bfa9-84eb251430fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:10:07.153233 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.153192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:07.153233 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.153231    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/759717e8-7b06-4ad1-bfa9-84eb251430fc-proxy-tls\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:07.153233 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.153243    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/759717e8-7b06-4ad1-bfa9-84eb251430fc-cabundle-cert\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:07.153519 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.153253    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cn9\" (UniqueName: \"kubernetes.io/projected/759717e8-7b06-4ad1-bfa9-84eb251430fc-kube-api-access-k7cn9\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:07.153519 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.153262    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/759717e8-7b06-4ad1-bfa9-84eb251430fc-kserve-provision-location\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:07.341597 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.341561    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:10:07.346964 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.346937    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cf00b2-predictor-7d877cbff-pdv62"]
Sep 18 20:10:07.793729 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:07.793671    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" path="/var/lib/kubelet/pods/759717e8-7b06-4ad1-bfa9-84eb251430fc/volumes"
Sep 18 20:10:08.015239 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015203    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z4v4l/must-gather-tn2dh"]
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015547    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015563    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015575    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015581    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015606    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015612    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015623    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015631    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015640    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kube-rbac-proxy"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015647    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kube-rbac-proxy"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015706    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kserve-container"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015713    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="759717e8-7b06-4ad1-bfa9-84eb251430fc" containerName="kube-rbac-proxy"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015720    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.015751 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.015726    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6785a832-68c9-4fe0-8fe5-d10cce162ad1" containerName="storage-initializer"
Sep 18 20:10:08.020093 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.020070    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.023605 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.023578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z4v4l\"/\"openshift-service-ca.crt\""
Sep 18 20:10:08.023760 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.023654    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z4v4l\"/\"kube-root-ca.crt\""
Sep 18 20:10:08.033395 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.033351    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z4v4l/must-gather-tn2dh"]
Sep 18 20:10:08.161357 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.161261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.161357 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.161315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tggd\" (UniqueName: \"kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.261743 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.261701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.261743 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.261745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tggd\" (UniqueName: \"kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.262079 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.262054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.271496 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.271464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tggd\" (UniqueName: \"kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd\") pod \"must-gather-tn2dh\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") " pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.338904 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.338848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:08.615049 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.615019    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z4v4l/must-gather-tn2dh"]
Sep 18 20:10:08.617545 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:10:08.617514    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod715c2071_9f1b_49fa_82b2_be408a03ac31.slice/crio-a2926917d2b2efddae73bfb4d675b498ca9ba43ed80ff5089cd5b3aae6523d5d WatchSource:0}: Error finding container a2926917d2b2efddae73bfb4d675b498ca9ba43ed80ff5089cd5b3aae6523d5d: Status 404 returned error can't find the container with id a2926917d2b2efddae73bfb4d675b498ca9ba43ed80ff5089cd5b3aae6523d5d
Sep 18 20:10:08.619440 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:08.619423    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:10:09.026713 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:09.026679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" event={"ID":"715c2071-9f1b-49fa-82b2-be408a03ac31","Type":"ContainerStarted","Data":"a2926917d2b2efddae73bfb4d675b498ca9ba43ed80ff5089cd5b3aae6523d5d"}
Sep 18 20:10:16.066194 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:16.066150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" event={"ID":"715c2071-9f1b-49fa-82b2-be408a03ac31","Type":"ContainerStarted","Data":"efcfa54aaa6134915d36b040e2b7ec297376f9519cf446fe033e31975ee2686e"}
Sep 18 20:10:16.066656 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:16.066209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" event={"ID":"715c2071-9f1b-49fa-82b2-be408a03ac31","Type":"ContainerStarted","Data":"991edebe106dbaada0c0ef818325079e6fb137bb37401793057e504584475f63"}
Sep 18 20:10:16.089849 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:16.089780    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" podStartSLOduration=2.724152852 podStartE2EDuration="9.089757513s" totalImagesPullingTime="6.365604661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:16.087132947 +0000 UTC m=+3348.462559239" watchObservedRunningTime="2026-09-18 20:10:16.089757513 +0000 UTC m=+3348.465183811"
Sep 18 20:10:31.933733 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:31.933139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:10:31.949150 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:31.934660    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:10:46.212909 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:46.212871    2583 generic.go:356] "Generic (PLEG): container finished" podID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerID="991edebe106dbaada0c0ef818325079e6fb137bb37401793057e504584475f63" exitCode=0
Sep 18 20:10:46.213381 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:46.212946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" event={"ID":"715c2071-9f1b-49fa-82b2-be408a03ac31","Type":"ContainerDied","Data":"991edebe106dbaada0c0ef818325079e6fb137bb37401793057e504584475f63"}
Sep 18 20:10:46.213381 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:46.213266    2583 scope.go:117] "RemoveContainer" containerID="991edebe106dbaada0c0ef818325079e6fb137bb37401793057e504584475f63"
Sep 18 20:10:46.448140 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:46.448099    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z4v4l_must-gather-tn2dh_715c2071-9f1b-49fa-82b2-be408a03ac31/gather/0.log"
Sep 18 20:10:50.642121 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:50.642083    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mqnsc_247b79f7-905d-4f9a-8dfc-078cea6f6aa7/global-pull-secret-syncer/0.log"
Sep 18 20:10:50.884156 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:50.884116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zhdrh_0b2692c8-adc0-4005-8214-5c90e6e12d4f/konnectivity-agent/0.log"
Sep 18 20:10:50.958173 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:50.958083    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-205.ec2.internal_91f5005043eff195cc3f3cd98b4b0f5f/haproxy/0.log"
Sep 18 20:10:52.019659 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.019613    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-z4v4l/must-gather-tn2dh"]
Sep 18 20:10:52.020167 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.019817    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-z4v4l/must-gather-tn2dh" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="copy" containerID="cri-o://efcfa54aaa6134915d36b040e2b7ec297376f9519cf446fe033e31975ee2686e" gracePeriod=2
Sep 18 20:10:52.025203 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.025177    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-z4v4l/must-gather-tn2dh"]
Sep 18 20:10:52.241627 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.241605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z4v4l_must-gather-tn2dh_715c2071-9f1b-49fa-82b2-be408a03ac31/copy/0.log"
Sep 18 20:10:52.242013 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.241991    2583 generic.go:356] "Generic (PLEG): container finished" podID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerID="efcfa54aaa6134915d36b040e2b7ec297376f9519cf446fe033e31975ee2686e" exitCode=143
Sep 18 20:10:52.254685 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.254663    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z4v4l_must-gather-tn2dh_715c2071-9f1b-49fa-82b2-be408a03ac31/copy/0.log"
Sep 18 20:10:52.256189 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.256167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:52.258708 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:52.258682    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tn2dh\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z4v4l\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:52.350919 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.350829    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output\") pod \"715c2071-9f1b-49fa-82b2-be408a03ac31\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") "
Sep 18 20:10:52.350919 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.350883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tggd\" (UniqueName: \"kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd\") pod \"715c2071-9f1b-49fa-82b2-be408a03ac31\" (UID: \"715c2071-9f1b-49fa-82b2-be408a03ac31\") "
Sep 18 20:10:52.352457 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.352417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "715c2071-9f1b-49fa-82b2-be408a03ac31" (UID: "715c2071-9f1b-49fa-82b2-be408a03ac31"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:10:52.353105 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.353082    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd" (OuterVolumeSpecName: "kube-api-access-6tggd") pod "715c2071-9f1b-49fa-82b2-be408a03ac31" (UID: "715c2071-9f1b-49fa-82b2-be408a03ac31"). InnerVolumeSpecName "kube-api-access-6tggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:10:52.452168 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.452121    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/715c2071-9f1b-49fa-82b2-be408a03ac31-must-gather-output\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:52.452168 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:52.452160    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tggd\" (UniqueName: \"kubernetes.io/projected/715c2071-9f1b-49fa-82b2-be408a03ac31-kube-api-access-6tggd\") on node \"ip-10-0-135-205.ec2.internal\" DevicePath \"\""
Sep 18 20:10:53.248471 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:53.248443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-z4v4l_must-gather-tn2dh_715c2071-9f1b-49fa-82b2-be408a03ac31/copy/0.log"
Sep 18 20:10:53.248875 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:53.248857    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:53.248961 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:53.248865    2583 scope.go:117] "RemoveContainer" containerID="efcfa54aaa6134915d36b040e2b7ec297376f9519cf446fe033e31975ee2686e"
Sep 18 20:10:53.252651 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:53.252615    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tn2dh\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z4v4l\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:53.260031 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:53.260002    2583 scope.go:117] "RemoveContainer" containerID="991edebe106dbaada0c0ef818325079e6fb137bb37401793057e504584475f63"
Sep 18 20:10:53.263350 ip-10-0-135-205 kubenswrapper[2583]: E0918 20:10:53.263321    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tn2dh\" is forbidden: User \"system:node:ip-10-0-135-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-z4v4l\": no relationship found between node 'ip-10-0-135-205.ec2.internal' and this object" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" pod="openshift-must-gather-z4v4l/must-gather-tn2dh"
Sep 18 20:10:53.791350 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:53.791317    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" path="/var/lib/kubelet/pods/715c2071-9f1b-49fa-82b2-be408a03ac31/volumes"
Sep 18 20:10:55.192799 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:55.192769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-mnhtw_7fb29efd-aee5-4e8e-918e-96d40d8c75bc/cluster-monitoring-operator/0.log"
Sep 18 20:10:55.430432 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:55.430340    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-b9zds_25ed0475-e04e-4064-939f-19eafd7748f0/monitoring-plugin/0.log"
Sep 18 20:10:55.489103 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:55.489075    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6r87q_4f027814-71ea-46c3-a90c-91550410847b/node-exporter/0.log"
Sep 18 20:10:55.525157 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:55.525128    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6r87q_4f027814-71ea-46c3-a90c-91550410847b/kube-rbac-proxy/0.log"
Sep 18 20:10:55.559823 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:55.559797    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6r87q_4f027814-71ea-46c3-a90c-91550410847b/init-textfile/0.log"
Sep 18 20:10:56.129715 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.129683    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/prometheus/0.log"
Sep 18 20:10:56.216604 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.216565    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/config-reloader/0.log"
Sep 18 20:10:56.314478 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.314445    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/thanos-sidecar/0.log"
Sep 18 20:10:56.454159 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.454068    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/kube-rbac-proxy-web/0.log"
Sep 18 20:10:56.503665 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.503636    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/kube-rbac-proxy/0.log"
Sep 18 20:10:56.560050 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.560017    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/kube-rbac-proxy-thanos/0.log"
Sep 18 20:10:56.638549 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.638520    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_96ea603c-cafe-4dbc-a915-ba779db8b4da/init-config-reloader/0.log"
Sep 18 20:10:56.707076 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.706993    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-xttd2_c4283428-426d-4c47-aa9f-8533caee3298/prometheus-operator/0.log"
Sep 18 20:10:56.772453 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:56.772424    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-xttd2_c4283428-426d-4c47-aa9f-8533caee3298/kube-rbac-proxy/0.log"
Sep 18 20:10:57.149151 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.149123    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/thanos-query/0.log"
Sep 18 20:10:57.211809 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.211782    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/kube-rbac-proxy-web/0.log"
Sep 18 20:10:57.243044 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.243017    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/kube-rbac-proxy/0.log"
Sep 18 20:10:57.273719 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.273626    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/prom-label-proxy/0.log"
Sep 18 20:10:57.309161 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.309123    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/kube-rbac-proxy-rules/0.log"
Sep 18 20:10:57.352753 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:57.352726    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848f7dcf97-xvbsb_0ce461fc-a0e5-44c2-b013-31ea73f469e1/kube-rbac-proxy-metrics/0.log"
Sep 18 20:10:58.192690 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.192652    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"]
Sep 18 20:10:58.193081 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193062    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="copy"
Sep 18 20:10:58.193171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193083    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="copy"
Sep 18 20:10:58.193171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193100    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="gather"
Sep 18 20:10:58.193171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="gather"
Sep 18 20:10:58.193323 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193180    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="copy"
Sep 18 20:10:58.193323 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.193195    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="715c2071-9f1b-49fa-82b2-be408a03ac31" containerName="gather"
Sep 18 20:10:58.197455 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.197424    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.201372 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.201330    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92dww\"/\"openshift-service-ca.crt\""
Sep 18 20:10:58.201499 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.201396    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92dww\"/\"kube-root-ca.crt\""
Sep 18 20:10:58.202599 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.202585    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-92dww\"/\"default-dockercfg-2fh4v\""
Sep 18 20:10:58.207770 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.207746    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"]
Sep 18 20:10:58.298096 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.298064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-lib-modules\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.298484 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.298105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-proc\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.298484 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.298148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-sys\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.298484 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.298192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-podres\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.298484 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.298248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdm6\" (UniqueName: \"kubernetes.io/projected/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-kube-api-access-fcdm6\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399282 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-proc\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-sys\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-podres\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdm6\" (UniqueName: \"kubernetes.io/projected/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-kube-api-access-fcdm6\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-proc\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-sys\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-podres\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399506 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-lib-modules\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.399829 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.399544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-lib-modules\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.409865 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.409836    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdm6\" (UniqueName: \"kubernetes.io/projected/8bfa87eb-04d9-49ca-98f7-4d615c4c44be-kube-api-access-fcdm6\") pod \"perf-node-gather-daemonset-lhs6m\" (UID: \"8bfa87eb-04d9-49ca-98f7-4d615c4c44be\") " pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.507897 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.507859    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:58.781758 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:58.781728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"]
Sep 18 20:10:58.784104 ip-10-0-135-205 kubenswrapper[2583]: W0918 20:10:58.784074    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bfa87eb_04d9_49ca_98f7_4d615c4c44be.slice/crio-989cd6304a82f11d45db861ee899e415747fa5ec52f557ed3f7570ab404dbfa1 WatchSource:0}: Error finding container 989cd6304a82f11d45db861ee899e415747fa5ec52f557ed3f7570ab404dbfa1: Status 404 returned error can't find the container with id 989cd6304a82f11d45db861ee899e415747fa5ec52f557ed3f7570ab404dbfa1
Sep 18 20:10:59.106034 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.105938    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/2.log"
Sep 18 20:10:59.113154 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.113127    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-wbklz_06e130ed-b5d1-47e6-ad81-26d0cc582900/console-operator/3.log"
Sep 18 20:10:59.275171 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.275126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m" event={"ID":"8bfa87eb-04d9-49ca-98f7-4d615c4c44be","Type":"ContainerStarted","Data":"20580ae21619d3b044e3576da95b48bbf0ba21f88a93286943e70916ef00c618"}
Sep 18 20:10:59.275402 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.275185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m" event={"ID":"8bfa87eb-04d9-49ca-98f7-4d615c4c44be","Type":"ContainerStarted","Data":"989cd6304a82f11d45db861ee899e415747fa5ec52f557ed3f7570ab404dbfa1"}
Sep 18 20:10:59.275402 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.275226    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:10:59.300825 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:10:59.300765    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m" podStartSLOduration=1.300747507 podStartE2EDuration="1.300747507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:59.299942888 +0000 UTC m=+3391.675369183" watchObservedRunningTime="2026-09-18 20:10:59.300747507 +0000 UTC m=+3391.676173799"
Sep 18 20:11:01.362635 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:01.362599    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4g8h7_22a1f40d-14f0-468d-83f8-d34b818e84ee/dns/0.log"
Sep 18 20:11:01.396323 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:01.396283    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4g8h7_22a1f40d-14f0-468d-83f8-d34b818e84ee/kube-rbac-proxy/0.log"
Sep 18 20:11:01.576376 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:01.576333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4zxvd_561e9854-ab29-47f0-b3e0-620407e4d964/dns-node-resolver/0.log"
Sep 18 20:11:02.294687 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:02.294654    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xh8cv_faf1d939-4798-4480-bb3d-ba0f6cd73c6a/node-ca/0.log"
Sep 18 20:11:03.115258 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:03.115222    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-565c89cdb6-9s5v4_bc5a596d-ba1c-4596-9f7a-d551506eaec6/router/0.log"
Sep 18 20:11:03.691458 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:03.691429    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwl5f_fdcc952a-f8d1-4a6c-bcdb-a1e16dd66e2a/serve-healthcheck-canary/0.log"
Sep 18 20:11:04.209603 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:04.209566    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-9kp6f_c8ae4017-1968-4419-8996-91642ebd40ae/insights-operator/0.log"
Sep 18 20:11:04.210638 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:04.210613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-9kp6f_c8ae4017-1968-4419-8996-91642ebd40ae/insights-operator/1.log"
Sep 18 20:11:04.390709 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:04.390676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzmkc_1215df9a-994e-45a1-86ec-9c48436b5718/kube-rbac-proxy/0.log"
Sep 18 20:11:04.431517 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:04.431492    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzmkc_1215df9a-994e-45a1-86ec-9c48436b5718/exporter/0.log"
Sep 18 20:11:04.495060 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:04.495034    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gzmkc_1215df9a-994e-45a1-86ec-9c48436b5718/extractor/0.log"
Sep 18 20:11:05.289016 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:05.288990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-92dww/perf-node-gather-daemonset-lhs6m"
Sep 18 20:11:08.181819 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.181785    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-sgdgh_56be5525-3861-4a4e-93eb-036a076af52b/s3-init/0.log"
Sep 18 20:11:08.213252 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.213162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-sgdgh_56be5525-3861-4a4e-93eb-036a076af52b/download-hf-model/0.log"
Sep 18 20:11:08.265040 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.265007    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-jj7lb_2aaba14b-4684-4767-a2f6-1033f8d533cd/s3-tls-init-custom/0.log"
Sep 18 20:11:08.289104 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.289069    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-jj7lb_2aaba14b-4684-4767-a2f6-1033f8d533cd/download-hf-model/0.log"
Sep 18 20:11:08.354183 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.354143    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-gnplz_842d9ef7-c2aa-4e79-92ee-5f04f21aa02c/s3-tls-init-serving/0.log"
Sep 18 20:11:08.453898 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:08.453873    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-gnplz_842d9ef7-c2aa-4e79-92ee-5f04f21aa02c/download-hf-model/0.log"
Sep 18 20:11:13.548661 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:13.548625    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-ff4fx_de32a182-70c4-4c37-9f9b-35c766d1ff4f/migrator/0.log"
Sep 18 20:11:13.576866 ip-10-0-135-205 kubenswrapper[2583]: I0918 20:11:13.576835    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-ff4fx_de32a182-70c4-4c37-9f9b-35c766d1ff4f/graceful-termination/0.log"